Igor Lins e Silva
133dfbfb41
Three tightly-coupled search-quality fixes for v3.3.3: 1. CLI `mempalace search` now routes through the same `_hybrid_rank` the MCP path already used. Drawers whose text contains every query term but embed as file-tree noise (directory listings, diffs, log fragments) were scoring cosine distance >= 1.0 — the display formula `max(0, 1 - dist)` then floored every result to `Match: 0.0`, with no way for the user to tell a lexical match from a total miss. BM25 catches these cleanly; the display surfaces both `cosine=` and `bm25=` so users see which component is firing. 2. Legacy-palace distance-metric warning. Palaces created before `hnsw:space=cosine` was consistently set silently use ChromaDB's default L2 metric, which breaks the cosine-similarity formula (L2 distances routinely exceed 1.0 on normalized 384-dim vectors). The search path now detects this at query time and prints a one-line notice pointing at `mempalace repair`. Only fires for legacy palaces; new palaces already set cosine correctly. 3. Invariant tests pinning `hnsw:space=cosine` on every collection- creation path — legacy `get_or_create_collection`, legacy `create_collection`, RFC 001 `get_collection(create=True)`, the public `palace.get_collection`, and a round-trip through reopen. Locks down the correctness that new-user palaces already have so a future refactor can't silently regress it. Also adds a `metadata` property to `ChromaCollection` so callers can read the underlying hnsw:space without reaching into `_collection`. Tests: - New regression: simulate three candidates at distance 1.5 (cosine=0), one containing query terms — must rank first with non-zero bm25. - New: legacy metric (empty or non-cosine) produces stderr warning. - New: correctly-configured palace produces no warning. - New: all five creation paths pin cosine metadata. All existing tests still pass.
83 lines
3.4 KiB
Python
83 lines
3.4 KiB
Python
"""Invariant tests: every ChromaDB collection-creation path must set
|
|
``hnsw:space=cosine``.
|
|
|
|
Reason: ChromaDB's default HNSW distance is L2 (Euclidean). Under L2,
|
|
the searcher's ``max(0, 1 - distance)`` similarity formula systematically
|
|
floors to 0 because L2 distances on normalized 384-dim vectors routinely
|
|
exceed 1.0 — users then see flat ``Match: 0.0`` across every result and
|
|
have no signal that their palace is broken.
|
|
|
|
This test file locks the invariant so a future refactor that drops the
|
|
``metadata={"hnsw:space": "cosine"}`` parameter from any creation path
|
|
gets caught at test time rather than silently degrading search quality.
|
|
"""
|
|
|
|
import tempfile
|
|
|
|
from mempalace.backends.chroma import ChromaBackend
|
|
from mempalace.palace import get_collection
|
|
|
|
|
|
EXPECTED_METRIC = "cosine"
|
|
|
|
|
|
def _assert_cosine(col, where: str) -> None:
|
|
meta = col.metadata if hasattr(col, "metadata") else col._collection.metadata
|
|
assert isinstance(meta, dict), f"{where}: expected metadata dict, got {meta!r}"
|
|
assert meta.get("hnsw:space") == EXPECTED_METRIC, (
|
|
f"{where}: expected hnsw:space={EXPECTED_METRIC!r}, got {meta!r}. "
|
|
"A collection without cosine metric will silently break the "
|
|
"similarity formula used by the searcher."
|
|
)
|
|
|
|
|
|
def test_legacy_get_or_create_collection_sets_cosine(tmp_path):
|
|
backend = ChromaBackend()
|
|
col = backend.get_or_create_collection(str(tmp_path), "mempalace_drawers")
|
|
_assert_cosine(col, "legacy get_or_create_collection")
|
|
|
|
|
|
def test_legacy_create_collection_sets_cosine(tmp_path):
|
|
backend = ChromaBackend()
|
|
col = backend.create_collection(str(tmp_path), "mempalace_drawers")
|
|
_assert_cosine(col, "legacy create_collection")
|
|
|
|
|
|
def test_new_get_collection_with_create_sets_cosine(tmp_path):
|
|
"""RFC 001 typed surface — ``get_collection(..., create=True)`` is the
|
|
path the miner + init flow take. Must also set cosine."""
|
|
backend = ChromaBackend()
|
|
col = backend.get_collection(str(tmp_path), "mempalace_drawers", create=True)
|
|
_assert_cosine(col, "get_collection(create=True)")
|
|
|
|
|
|
def test_palace_module_get_collection_sets_cosine(tmp_path):
|
|
"""The public ``mempalace.palace.get_collection`` is what most callers
|
|
use. Must produce cosine palaces."""
|
|
col = get_collection(str(tmp_path), "mempalace_drawers", create=True)
|
|
_assert_cosine(col, "palace.get_collection(create=True)")
|
|
|
|
|
|
def test_reopening_cosine_palace_preserves_metric(tmp_path):
|
|
"""Opening a previously-created cosine palace (create=False) must
|
|
still expose the cosine metadata — catches any regression where
|
|
reopening drops or overwrites metadata."""
|
|
backend = ChromaBackend()
|
|
backend.create_collection(str(tmp_path), "mempalace_drawers")
|
|
# Fresh backend simulates a process restart
|
|
backend2 = ChromaBackend()
|
|
col = backend2.get_collection(str(tmp_path), "mempalace_drawers", create=False)
|
|
_assert_cosine(col, "re-opened palace")
|
|
|
|
|
|
def test_fresh_palace_via_full_stack_gets_cosine():
|
|
"""End-to-end: build a palace with the public API the way a new user
|
|
would, confirm the resulting collection uses cosine distance."""
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
col = get_collection(tmp, "mempalace_drawers", create=True)
|
|
_assert_cosine(col, "full-stack new palace")
|
|
|
|
# And the closets collection too
|
|
closets = get_collection(tmp, "mempalace_closets", create=True)
|
|
_assert_cosine(closets, "full-stack new closets")
|