Mirror the pagination pattern PR #851 landed in miner.py:status().
A single drawers_col.get(limit=total, ...) on palaces larger than
SQLite's SQLITE_MAX_VARIABLE_NUMBER (32766) crashes inside chromadb.
Fetch drawers in batch_size=5000 chunks, stepping offset until the
collection is drained. by_source aggregation semantics are preserved
exactly — grouping, wing filter, meta capture all unchanged.
Closes#1073. Related: #802, #850, #1016.
Brings in PR #793 (optional LLM-based closet regeneration via
user-configured OpenAI-compatible endpoint) and PR #795 (hybrid
closet+drawer search — closets boost, never gate). Stack: #784 → #788
→ #789 → #790 → #791 → #792 → #793 (+ #795).
Findings hardened on our side
─────────────────────────────
1) closet_llm.regenerate_closets didn't use the blessed palace helpers.
Before:
* manual closets_col.get(where=...) + .delete(ids=...) with a
silent ``except Exception: pass`` around both — if the purge
failed, pre-existing regex closets survived alongside fresh LLM
closets, giving the searcher double hits for the same source.
* ``source.split('/')[-1][:30]`` to build the closet_id — quietly
wrong on Windows paths (``C:\\proj\\a.md`` has no ``/``, so the
whole string ends up in the ID).
* no mine_lock around purge+upsert — a concurrent regex rebuild of
the same source could interleave with our purge and leave a mix
of regex and LLM pointers.
* no ``normalize_version`` stamp on the LLM closets — the miner's
stale-version gate would treat them as leftovers from an older
schema and rebuild over them on the next mine.
After: routes through ``purge_file_closets`` + ``mine_lock`` +
``os.path.basename`` + ``NORMALIZE_VERSION`` stamp. Regression tests
cover each.
2) searcher.search_memories was still closet-first.
PR #795 merged into #793's head to fix the recall regression
documented in that PR (R@1 0.25 on narrative content vs. 0.42
baseline). The hybrid design makes closets a ranking boost rather
than a gate: drawers are always queried at the floor, and matching
closet hits (rank 0-4 within CLOSET_DISTANCE_CAP=1.5) add a boost
of 0.40/0.25/0.15/0.08/0.04 to the effective distance.
Merged to take the incoming hybrid design, with two cleanups:
* kept the ``_expand_with_neighbors`` / ``_extract_drawer_ids_from_closet``
helpers as separately-tested utilities (still imported by tests
and future callers);
* replaced the fragile ``source_file.endswith(basename)`` reverse-
lookup in the enrichment step with internal ``_source_file_full``
/ ``_chunk_index`` fields stripped before return, so enrichment
doesn't silently pick the wrong path when two sources share a
basename across directories;
* drawer-grep enrichment now sorts by ``chunk_index`` before
neighbor expansion, so ``best_idx ± 1`` corresponds to actual
document order rather than whatever order Chroma returned.
3) Closet-first tests in test_closets.py (``TestSearchMemoriesClosetFirst``,
end-to-end ``test_closet_first_search_includes_drawer_index_and_total``)
pinned contracts that the hybrid path now violates (``matched_via``
went from ``"closet"`` to ``"drawer+closet"``). Rewrote them around
the new invariant: direct drawers are always the floor, closet
agreement flips the hit's matched_via and exposes closet_preview.
Verification
────────────
* 805/805 pass under ``uv run pytest tests/ -v --ignore=tests/benchmarks``
(13 new tests from PR #793 + 5 from PR #795 + 2 new regressions for
the closet_llm hardening + the rewritten hybrid assertions in
test_closets.py).
* CI-pinned ruff 0.4.x clean on ``mempalace/`` + ``tests/`` (check +
format both pass).
* No new deps — closet_llm.py still uses stdlib ``urllib.request`` per
the PR's "zero new dependencies" promise.
Co-Authored-By: MSL <232237854+milla-jovovich@users.noreply.github.com>
Adds mempalace/closet_llm.py as an OPTIONAL path for richer closet
generation. Regex closets remain the default and cover the local-first
promise; users who want LLM-quality topics can bring their own endpoint.
Configuration (env or CLI flag):
LLM_ENDPOINT — OpenAI-compatible base URL (required)
LLM_KEY — bearer token (optional; local inference skips this)
LLM_MODEL — model name (required)
Works with Ollama, vLLM, llama.cpp servers, OpenAI, OpenRouter, and any
other provider that speaks OpenAI-compatible /chat/completions. Zero new
dependencies — uses stdlib urllib.
Replaces the original Anthropic-SDK-hardcoded version of this module
from Milla's branch (commit 935f657). Same prompt, same parsing, same
regenerate_closets flow; only the transport was generalised so the
feature doesn't lock users into a specific vendor or require API keys
for core memory operations (CLAUDE.md, "Local-first, zero API").
Includes 13 unit tests covering config resolution, request shape,
auth-header omission when no key is set, code-fence stripping, and
missing-config error path. All mocked — zero network calls in tests.
Co-Authored-By: MSL <232237854+milla-jovovich@users.noreply.github.com>
sha2fiddy