The pre-existing test_maybe_run_mine_prompt_declined_prints_hint
asserted the bare unquoted form `mempalace mine {tmp_path}`. After
the production code switched to shlex.quote on the resume hint, this
passed on Linux/macOS (POSIX paths have no characters that trigger
quoting) but failed on Windows where backslashes always get wrapped
in single quotes.
Mirror the production code in the assertion via shlex.quote so it's
portable across platforms; do the same for the two new
spaces-in-path tests for consistency.
The "Skipped. Run mempalace mine <dir>" hint after declining the init
prompt and the "Re-run mempalace mine <dir> to resume" hint after a
Ctrl-C interruption both interpolated project_dir without shell-quoting.
A path containing spaces or metacharacters produced a copy-paste-broken
command.
Both spots now use shlex.quote(project_dir). Adds regression tests
covering each hint with a path that contains a space.
Reviewer feedback on the previous commit flagged two real problems:
1. Overloading --yes to also auto-mine was a silent behaviour change for
scripted callers. Today --yes only auto-accepts entities — making it
ALSO trigger a multi-minute ChromaDB write breaks every script that
currently runs `mempalace init --yes <dir>` for the fast non-interactive
entity path. Add a separate `--auto-mine` flag instead. Combinations:
mempalace init --yes <dir> # entities auto, STILL prompt mine
mempalace init --auto-mine <dir> # prompt entities, skip mine prompt
mempalace init --yes --auto-mine <dir> # fully non-interactive
--yes behaviour is now identical to pre-PR.
2. The mine prompt was firing without telling the user how big the job
was. On a real corpus mine takes minutes-to-tens-of-minutes; hitting
Enter on default-Y with no size cue is a footgun. Show a one-line
estimate computed from scan_project (the same walk we hand into mine)
BEFORE the prompt:
~423 files (~12 MB) would be mined into this palace.
Mine this directory now? [Y/n]
The estimate uses a single corpus walk: scan_project's output is
passed into mine() via a new optional files= kwarg, so we never walk
the tree twice.
Tests: replaced the old "--yes auto-mines" assertion with a regression
guard that --yes alone STILL prompts; added coverage for --auto-mine
alone, --yes --auto-mine together, and the pre-prompt estimate line.
`mempalace init` now ends with a `Mine this directory now? [Y/n]`
prompt and runs `mine()` in-process when accepted; `--yes` skips the
prompt and auto-mines for non-interactive callers. Declining prints
the resume command. Removes the "remember to type the next command"
friction since rooms + entities just got set up.
`mempalace mine` now wraps its main loop in `try / except
KeyboardInterrupt` and prints `files_processed`, `drawers_filed`, and
`last_file` before exiting with code 130 on Ctrl-C. Re-mining is safe
because deterministic drawer IDs make the upsert idempotent. The
hooks PID lock at `~/.mempalace/hook_state/mine.pid` is now actively
removed in a `finally` when its entry points at us, on clean exit,
error, or interrupt — preventing the next hook fire from briefly
waiting on a stale PID.
Closes#1181, #1182.
`test_fresh_palace_via_full_stack_gets_cosine` used `tempfile.Temporary-
Directory()` as a context manager, which tries to delete the temp path
on exit. On Windows, ChromaDB still holds SQLite file handles to
`chroma.sqlite3` when the context closes, producing:
PermissionError: [WinError 32] The process cannot access the file
because it is being used by another process: '...\\chroma.sqlite3'
NotADirectoryError: [WinError 267] The directory name is invalid
Other tests in the same file use pytest's `tmp_path` fixture, which
defers cleanup to session end (when the process is exiting and the
file-lock contention is moot). Align this one with the rest of the
file.
CLAUDE.md already documents the 80% Windows coverage allowance due to
"ChromaDB file lock cleanup" — the fix is to stop fighting the lock.
Three tightly-coupled search-quality fixes for v3.3.3:
1. CLI `mempalace search` now routes through the same `_hybrid_rank`
the MCP path already used. Drawers whose text contains every query
term but embed as file-tree noise (directory listings, diffs, log
fragments) were scoring cosine distance >= 1.0 — the display formula
`max(0, 1 - dist)` then floored every result to `Match: 0.0`, with
no way for the user to tell a lexical match from a total miss. BM25
catches these cleanly; the display surfaces both `cosine=` and
`bm25=` so users see which component is firing.
2. Legacy-palace distance-metric warning. Palaces created before
`hnsw:space=cosine` was consistently set silently use ChromaDB's
default L2 metric, which breaks the cosine-similarity formula (L2
distances routinely exceed 1.0 on normalized 384-dim vectors). The
search path now detects this at query time and prints a one-line
notice pointing at `mempalace repair`. Only fires for legacy
palaces; new palaces already set cosine correctly.
3. Invariant tests pinning `hnsw:space=cosine` on every collection-
creation path — legacy `get_or_create_collection`, legacy
`create_collection`, RFC 001 `get_collection(create=True)`, the
public `palace.get_collection`, and a round-trip through reopen.
Locks down the correctness that new-user palaces already have so a
future refactor can't silently regress it.
Also adds a `metadata` property to `ChromaCollection` so callers can
read the underlying hnsw:space without reaching into `_collection`.
Tests:
- New regression: simulate three candidates at distance 1.5 (cosine=0),
one containing query terms — must rank first with non-zero bm25.
- New: legacy metric (empty or non-cosine) produces stderr warning.
- New: correctly-configured palace produces no warning.
- New: all five creation paths pin cosine metadata.
All existing tests still pass.
Previously a cross-wing topic tunnel for "Angular" stored the room as
"Angular" — colliding with a wing's literal folder-derived "Angular" room
at follow_tunnels/list_tunnels read time, and exposing raw topic strings
(which may contain characters rejected by sanitize_name) to the MCP
surface.
Topic tunnels now store their room as "topic:<original-casing>" and carry
kind="topic" on the stored dict. Explicit tunnels get kind="explicit"
(default). follow_tunnels("wing", "Angular") on a literal Angular room
no longer surfaces topic connections for the same name, and any LLM
scanning list_tunnels has a visible discriminator.
When two wings have one or more confirmed TOPIC labels in common, the
miner now drops a symmetric tunnel between them at mine time so the
palace graph reflects shared themes (frameworks, vendors, recurring
concepts).
- llm_refine: TOPIC label routes to a dedicated `topics` bucket so the
signal survives confirmation instead of getting collapsed into
`uncertain` and dropped.
- entity_detector / project_scanner: bucket plumbed through the
detection pipeline; `confirm_entities` returns confirmed topics
alongside people/projects.
- miner.add_to_known_entities: optional `wing` parameter records the
confirmed topics under `topics_by_wing` in
`~/.mempalace/known_entities.json`. Wing names do NOT leak into the
flat known-name set used by drawer-tagging.
- palace_graph: `compute_topic_tunnels` and `topic_tunnels_for_wing`
create symmetric tunnels via the existing `create_tunnel` API so they
share dedup and persistence with explicit tunnels.
- miner.mine: post-file-loop pass calls `topic_tunnels_for_wing` for
the freshly-mined wing. Failures are logged but never abort the mine.
- config: `topic_tunnel_min_count` knob (env
`MEMPALACE_TOPIC_TUNNEL_MIN_COUNT` or `~/.mempalace/config.json`),
default 1.
Tests cover topic persistence through init->mine, tunnel creation when
wings share a topic, no tunnel below threshold, cross-wing tunnel
retrieval via `list_tunnels`, dedup on recompute, case-insensitive
overlap, and the end-to-end mine-time wiring.
Out of scope for this PR (called out in the PR body): manifest-
dependency overlap, per-topic allow/deny lists, search-result surfacing.
The miner upserted one drawer per ChromaDB call, paying tokenizer +
ONNX session setup per chunk. The embedding device was CPU-only because
no EmbeddingFunction was ever wired through the backend.
Two changes, each a speedup in its own right; stacked they give ~10x
end-to-end on a medium corpus (20 files, 568 drawers):
1. Batched upsert. `process_file` and `_file_chunks_locked` now collect
all chunks of a file into a single `collection.upsert(...)` so the
embedding model runs one forward pass per file instead of N.
2. Hardware-accelerated embedding function. New `mempalace/embedding.py`
wraps `ONNXMiniLM_L6_V2` with configurable `preferred_providers`.
`MEMPALACE_EMBEDDING_DEVICE` (or `embedding_device` in config.json)
selects auto / cpu / cuda / coreml / dml. Unavailable accelerators
log a warning and fall back to CPU.
The factory subclasses `ONNXMiniLM_L6_V2` and spoofs its `name()` to
`"default"` so the persisted EF identity matches existing palaces
created with ChromaDB's bare `DefaultEmbeddingFunction` -- same
model, same 384-dim vectors, no rebuild needed when turning GPU on.
`ChromaBackend.get_collection` / `create_collection` now pass the
resolved EF on every call so miner writes and searcher reads agree.
Benchmarks (i9-12900KF + RTX 3090, medium scenario, 568 drawers):
per-chunk + CPU 19.77s · 29 drw/s (baseline)
batched + CPU 8.07s · 70 drw/s (2.4x)
batched + CUDA 2.15s · 264 drw/s (9.2x)
Reproducible via `benchmarks/mine_bench.py`.
Install paths:
pip install mempalace[gpu] # NVIDIA CUDA
pip install mempalace[dml] # DirectML (Windows)
pip install mempalace[coreml] # macOS Neural Engine
Mine header now prints `Device: cpu|cuda|...` so users can confirm the
accelerator engaged.
Adds entries to the 3.3.3 section for the work that landed via #1148,
#1150, #1157, and #1175 (rescued from stacked feature branches into
develop via #1175). Without these entries the 3.3.3 release notes on
main would advertise only the hook/diary/search fixes that made it to
develop through the first direct merge.
Covers:
- Manifest + git-author entity detection (#1148)
- Regex detector accuracy improvements (#1148)
- Optional --llm classification with Ollama / openai-compat / Anthropic
provider abstraction and interactive UX (#1150)
- Claude Code conversation scanner (#1150)
- Init → miner registry wire-up so confirmed entities actually reach
drawer metadata tagging (#1157)
- Case-insensitive project dedup across all sources (#1175)
- `mempalace mine` skips the generated entities.json artifact
`discover_entities` was deduping the convo_scanner results against the
manifest/git scan with a case-sensitive key, while every other dedup
path in the pipeline (`_merge_detected`, `miner.add_to_known_entities`)
uses case-insensitive matching. A project named `foo` in a manifest
plus `Foo` as a Claude Code `cwd` variant would surface as two review
entries instead of collapsing to one.
Fix keys `by_name` by `name.lower()` while preserving the first-seen
casing, matching the rest of the pipeline. Flagged by Copilot on #1175.
Regression test asserts a manifest project + a CamelCase-variant convo
cwd for the same real project collapse to one entry.
#1148, #1150, and #1157 were reviewed and merged on GitHub, but the two
stacked children landed on their parent feature branches (now stale)
rather than on develop. Only #1148's commits reached develop via the
direct merge. Release PR #1159 (develop → main for v3.3.3) is therefore
missing the LLM refinement, Claude-conversation scanner, and miner-
registry wire-up that were ostensibly part of the release.
This merge brings the stale `feat/llm-entity-refine` branch (which
contains the rolled-up merge commit for #1157 → #1150 → everything
below) into develop so the release tag includes it.
No code changes here — only history recovery.
Windows 8.3 short paths legitimately contain tildes (e.g. the CI runner's
USERPROFILE resolves to C:\Users\RUNNER~1\...), so asserting "~" is absent
from the expanded path fails on Windows even when expanduser worked
correctly. The equality check against os.path.abspath(os.path.expanduser())
is authoritative; drop the redundant absence heuristic.
The new abspath+expanduser normalization means /env/palace no longer
round-trips literally on Windows (abspath prepends the current drive,
producing D:\env\palace). Rewrite the env-var tests to compare against
os.path.abspath(os.path.expanduser(raw)) instead of hardcoded Unix
strings, and build raw paths with os.path.join so backslash-vs-slash
differences don't leak into assertions. Covers test_env_override, the
three new tests, and the legacy-alias test in test_config_extra.
MEMPALACE_PALACE_PATH (and legacy MEMPAL_PALACE_PATH) read from the
environment was returned as-is from Config.palace_path, while the
sibling --palace CLI path gets os.path.abspath() applied at
mcp_server.py:62. That inconsistency means env-var callers can end
up with literal '~' or unresolved '..' segments in the path, which
(a) breaks user intuition and (b) lets a caller who can set env vars
on the target user's session redirect palace storage to an
unexpected location.
Apply os.path.abspath(os.path.expanduser(...)) to the env-var branch
so both code paths converge on the same resolved absolute path.
Closes#1163
The init step's output was a dead file. miner.py has always read
`~/.mempalace/known_entities.json` to tag drawer metadata with
recognized names, but nothing ever wrote it — so init's careful
manifest + git + LLM detection work stopped at `<project>/entities.json`
and never reached the path that actually uses it.
Measured delta on a representative prose snippet (eight sentences
mentioning six real people and four real projects):
- Empty registry: 0 entities recognized (multi-word names fail the
frequency threshold; lowercase/hyphenated project names don't match
the CamelCase regex).
- Registry populated by init: 12 entities recognized (all correct, zero
false positives).
Every recognized name becomes a semicolon-separated metadata tag on the
drawer, which ChromaDB uses for entity-filtered search.
Implementation:
- `miner.add_to_known_entities({category: [names]})` reads the existing
registry, unions each category (case-insensitively, preserving first-
seen casing), and writes back. The function is tolerant of the two
on-disk shapes miner already supports: list of names, or dict mapping
name → code (dialect-style). In the dict case new names are added as
keys with `None` values so existing codes aren't overwritten.
- Invalidates the in-process mtime cache so same-process callers
(`cmd_init` → `cmd_mine` in one run) see the write immediately.
- Writes with `ensure_ascii=False` so non-ASCII names (Gergő Móricz,
Arturo Domínguez, etc.) stay readable on disk.
- Chmods 0o600 — the registry mirrors confirm-step PII from the user's
git authors and local paths.
cmd_init now calls this at the end of the confirm-entities step, after
the per-project `entities.json` is written (which is kept as an audit
trail the user can inspect or hand-edit). The per-project file is still
excluded from mining via `SKIP_FILENAMES` from the earlier fix.
17 new tests cover: fresh-file creation, list-category union, case-
insensitive dedup, preservation of untouched categories, dict-format
registries, malformed/non-dict file recovery, cache invalidation,
unicode round-trip, and an end-to-end verification that the miner's
`_extract_entities_for_metadata` picks up every registered name.
Addresses issues found while reviewing the initial phase-2 implementation
against real data:
**Bug: uncertain bucket starved from the LLM.**
`discover_entities` was dropping the regex-uncertain bucket whenever real
git/manifest signal existed — which is exactly when `--llm` is most useful
for cleaning up prose noise. The uncertain candidates never reached the
refinement step. Fixed: only drop when `llm_provider is None`.
**Context collection: word boundaries, not substring.**
`_collect_contexts` used substring matching on lower-cased lines, so the
name "Go" matched "good", "going", "forgot". Switched to a
`(?<!\w)…(?!\w)` regex so short names only match at token boundaries.
**Authoritative-source detection replaces confidence threshold.**
Previously the refinement step skipped entries with `confidence >= 0.95`
to avoid second-guessing manifest-backed projects. That threshold was
fragile — the regex detector produces 0.99 confidence for things like
`code file reference (5x)` on framework names (OpenAPI, etc.), so those
skipped the LLM despite being regex-only noise. New helpers
`_is_authoritative_person` / `_is_authoritative_project` look at the
actual signal strings (commits, package.json, etc.) to decide.
**Now also refines regex-derived people.**
After #1148's high-pronoun-signal fix, the regex detector can promote
non-people to the `people` bucket (e.g. a capitalized common noun that
happened to appear near pronouns). The LLM now gets a chance to clean
those up, while git-authored people are still skipped.
**Robust JSON extraction.**
Small local models routinely wrap JSON output in prose ("Sure, here's
the classification: {…}"). The previous code-fence stripper failed on
that. `_extract_json_candidates` now does balanced-bracket extraction
with string-aware quote handling, so it recovers JSON from:
- raw responses
- markdown fenced blocks
- JSON embedded inside surrounding text
- multiple candidate objects/arrays
**Prompt guidance for frameworks vs user projects.**
Added an explicit instruction: frameworks, runtimes, APIs, cloud
services, and third-party vendors (Angular, OpenAPI, Terraform, Bun,
Google, etc.) are TOPIC unless the context clearly says it's the user's
own codebase. Directly addresses a false-positive pattern observed
during dev runs.
**Defensive mtime.**
`convo_scanner._safe_mtime` catches OSError during `stat()` — permission
changes, filesystem races, broken symlinks — and sorts the affected file
to the end of the newest-first order rather than crashing the scan.
**Cosmetic:** merged two adjacent f-strings on the same line in
`backends/chroma.py` and `llm_client.py` (no behaviour change).
15 new tests cover the OSError fallback, word-boundary matching, JSON
extraction variants, authoritative-source helpers, refining high-
confidence regex projects, and end-to-end LLM refinement preserving the
uncertain bucket.
Extends the init orchestrator to consume two new signal sources:
1. Claude Code conversation dirs: when the target is a
`~/.claude/projects/` root, convo_scanner contributes ProjectInfo
entries alongside the git/manifest projects. Dedup is by name,
preferring the entry with more user-authored activity.
2. Optional LLM refinement: when --llm is passed, discover_entities
constructs the provider, validates availability, and runs
llm_refine.refine_entities on the merged candidates. Status
summary (reclassified / dropped / cancelled / batch errors)
prints to stderr.
New init flags (opt-in, default remains zero-API):
- --llm: enable refinement
- --llm-provider: ollama (default) | openai-compat | anthropic
- --llm-model: default gemma4:e4b for Ollama
- --llm-endpoint: URL (required for openai-compat)
- --llm-api-key: falls back to env ($ANTHROPIC_API_KEY or
$OPENAI_API_KEY depending on provider)
Provider check_available runs before the scan, so the user sees an
immediate error ("Run: ollama pull <model>" or "ANTHROPIC_API_KEY not
set") rather than a mid-scan failure.
Takes the candidate set produced by phase-1 detection (manifests, git
authors, regex on prose) and asks an LLM to reclassify each candidate
as PERSON / PROJECT / TOPIC / COMMON_WORD / AMBIGUOUS.
Scale approach: never feed the raw corpus to the LLM. For each
candidate, collect up to 3 context lines from sampled prose, cap each
at 240 chars, batch 25 candidates per call. Keeps total input around
50-100K tokens even on large corpora and completes in a few minutes
on a 4B local model.
Interactive UX:
- Stderr progress bar with the current candidate name, updates
per-batch.
- Ctrl-C interrupts cleanly: returns a RefineResult with
`cancelled=True` and whatever was classified before the interrupt.
The partial result is safe to pass straight to confirm_entities.
- Per-batch errors (transport, parse) are recorded in `errors` and
don't abort the whole run.
Refinement scope: only `uncertain` and low-confidence `projects`
entries are sent. Manifest-backed projects (conf >= 0.95) and git-
authored people are already authoritative and skip the LLM.
Response parser is defensive — accepts `label` or `type` keys,
lowercase/uppercase variants, top-level list or wrapped object, and
strips markdown code fences. Unknown labels become AMBIGUOUS so the
user reviews them rather than silently accepting a bad classification.
`collect_corpus_text` provides a simple stratified prose sampler
(recent first, capped per-file) so callers don't need to build their
own corpus window.
28 tests with a FakeProvider (no network). Covers context collection,
prompt building, response parsing variants, classification apply,
end-to-end refine, and Ctrl-C partial-result behavior.
Three providers cover the useful space while keeping the zero-API
default:
- `ollama` (default): local models via http://localhost:11434. Works
fully offline. Tag-matching check accepts both `model` and
`model:latest` forms.
- `openai-compat`: any /v1/chat/completions endpoint. Covers
OpenRouter, LM Studio, llama.cpp server, vLLM, Groq, Together,
Fireworks, and most self-hosted frameworks. API key falls back to
$OPENAI_API_KEY. Endpoint normalization is forgiving about trailing
`/v1`.
- `anthropic`: Messages API v2023-06-01. API key falls back to
$ANTHROPIC_API_KEY. Concatenates multi-block text responses.
JSON mode is normalized across providers — Ollama uses
`format: "json"`, OpenAI-compat uses `response_format`, Anthropic uses
prompt-level instruction. Callers request JSON once; this module
handles the provider-specific plumbing.
No external SDK dependency; stdlib `urllib` throughout. HTTP errors
are wrapped into a single `LLMError` class so callers don't need to
distinguish transport, auth, and parse failures at the call site.
26 tests, all with mocked HTTP — suite runs offline with no real
provider required.
Claude Code stores sessions under `~/.claude/projects/<slug>/<id>.jsonl`
where `<slug>` is the original CWD with `/` replaced by `-`. That
encoding is lossy — can't distinguish `foo-bar` (one segment) from
`foo/bar` (two) — so slug-decoding alone produces wrong names for any
hyphenated project.
Fortunately, every message record carries a `cwd` field with the true
path. This scanner reads one record per session to recover the
accurate project name deterministically, falling back to slug-decoding
only if the JSONL is malformed or empty.
Output shape matches project_scanner.ProjectInfo so the discover
orchestrator can union results across sources. Session count doubles
as a density signal for ranking.
22 unit tests cover: root detection, cwd extraction with malformed
input tolerance, fallback slug decoding, name resolution using the
newest session (so renames win), and dedup when two encoded dirs
resolve to the same project.
`tomllib` is stdlib only in Python 3.11+. On Python 3.9/3.10 (and the
macOS runner) the scanner's toml parsing returned empty, so manifest
lookups for `pyproject.toml` / `Cargo.toml` produced no name. CI
surfaced this via 4 test_project_scanner.py failures on the 3.9 matrix.
Add `tomli>=2.0.0` as a conditional dependency for `python_version <
'3.11'` and fall back to it in `project_scanner.py`. The project still
declares `requires-python = ">=3.9"` so the fallback is the correct
shape.
Igor Lins e Silva