fix(search): BM25 hybrid rerank, legacy-metric warning, invariant tests

Three tightly-coupled search-quality fixes for v3.3.3:

1. CLI `mempalace search` now routes through the same `_hybrid_rank`
   the MCP path already used. Drawers whose text contains every query
   term but embed as file-tree noise (directory listings, diffs, log
   fragments) were scoring cosine distance >= 1.0 — the display formula
   `max(0, 1 - dist)` then floored every result to `Match: 0.0`, with
   no way for the user to tell a lexical match from a total miss. BM25
   catches these cleanly; the display surfaces both `cosine=` and
   `bm25=` so users see which component is firing.

2. Legacy-palace distance-metric warning. Palaces created before
   `hnsw:space=cosine` was consistently set silently use ChromaDB's
   default L2 metric, which breaks the cosine-similarity formula (L2
   distances routinely exceed 1.0 on normalized 384-dim vectors). The
   search path now detects this at query time and prints a one-line
   notice pointing at `mempalace repair`. Only fires for legacy
   palaces; new palaces already set cosine correctly.

3. Invariant tests pinning `hnsw:space=cosine` on every collection-
   creation path — legacy `get_or_create_collection`, legacy
   `create_collection`, RFC 001 `get_collection(create=True)`, the
   public `palace.get_collection`, and a round-trip through reopen.
   Locks down the correctness that new-user palaces already have so a
   future refactor can't silently regress it.

Also adds a `metadata` property to `ChromaCollection` so callers can
read the underlying hnsw:space without reaching into `_collection`.

Tests:
- New regression: simulate three candidates at distance 1.5 (cosine=0),
  one containing query terms — must rank first with non-zero bm25.
- New: legacy metric (empty or non-cosine) produces stderr warning.
- New: correctly-configured palace produces no warning.
- New: all five creation paths pin cosine metadata.

All existing tests still pass.
This commit is contained in:
Igor Lins e Silva
2026-04-24 18:50:28 -03:00
parent b9e41286fa
commit 133dfbfb41
5 changed files with 238 additions and 5 deletions
+82
View File
@@ -0,0 +1,82 @@
"""Invariant tests: every ChromaDB collection-creation path must set
``hnsw:space=cosine``.
Reason: ChromaDB's default HNSW distance is L2 (Euclidean). Under L2,
the searcher's ``max(0, 1 - distance)`` similarity formula systematically
floors to 0 because L2 distances on normalized 384-dim vectors routinely
exceed 1.0 — users then see flat ``Match: 0.0`` across every result and
have no signal that their palace is broken.
This test file locks the invariant so a future refactor that drops the
``metadata={"hnsw:space": "cosine"}`` parameter from any creation path
gets caught at test time rather than silently degrading search quality.
"""
import tempfile
from mempalace.backends.chroma import ChromaBackend
from mempalace.palace import get_collection
EXPECTED_METRIC = "cosine"
def _assert_cosine(col, where: str) -> None:
meta = col.metadata if hasattr(col, "metadata") else col._collection.metadata
assert isinstance(meta, dict), f"{where}: expected metadata dict, got {meta!r}"
assert meta.get("hnsw:space") == EXPECTED_METRIC, (
f"{where}: expected hnsw:space={EXPECTED_METRIC!r}, got {meta!r}. "
"A collection without cosine metric will silently break the "
"similarity formula used by the searcher."
)
def test_legacy_get_or_create_collection_sets_cosine(tmp_path):
backend = ChromaBackend()
col = backend.get_or_create_collection(str(tmp_path), "mempalace_drawers")
_assert_cosine(col, "legacy get_or_create_collection")
def test_legacy_create_collection_sets_cosine(tmp_path):
backend = ChromaBackend()
col = backend.create_collection(str(tmp_path), "mempalace_drawers")
_assert_cosine(col, "legacy create_collection")
def test_new_get_collection_with_create_sets_cosine(tmp_path):
"""RFC 001 typed surface — ``get_collection(..., create=True)`` is the
path the miner + init flow take. Must also set cosine."""
backend = ChromaBackend()
col = backend.get_collection(str(tmp_path), "mempalace_drawers", create=True)
_assert_cosine(col, "get_collection(create=True)")
def test_palace_module_get_collection_sets_cosine(tmp_path):
"""The public ``mempalace.palace.get_collection`` is what most callers
use. Must produce cosine palaces."""
col = get_collection(str(tmp_path), "mempalace_drawers", create=True)
_assert_cosine(col, "palace.get_collection(create=True)")
def test_reopening_cosine_palace_preserves_metric(tmp_path):
"""Opening a previously-created cosine palace (create=False) must
still expose the cosine metadata — catches any regression where
reopening drops or overwrites metadata."""
backend = ChromaBackend()
backend.create_collection(str(tmp_path), "mempalace_drawers")
# Fresh backend simulates a process restart
backend2 = ChromaBackend()
col = backend2.get_collection(str(tmp_path), "mempalace_drawers", create=False)
_assert_cosine(col, "re-opened palace")
def test_fresh_palace_via_full_stack_gets_cosine():
"""End-to-end: build a palace with the public API the way a new user
would, confirm the resulting collection uses cosine distance."""
with tempfile.TemporaryDirectory() as tmp:
col = get_collection(tmp, "mempalace_drawers", create=True)
_assert_cosine(col, "full-stack new palace")
# And the closets collection too
closets = get_collection(tmp, "mempalace_closets", create=True)
_assert_cosine(closets, "full-stack new closets")