The list_drawers response only included count (current page size) with
no total field, making it impossible for callers to know when pagination
is exhausted. A page returning count == limit is ambiguous — it could
be the last exact-fit page or there could be more results.
Add a total field that reports the full number of matching drawers.
For unfiltered requests this uses col.count(); for filtered requests
(wing/room) it uses a lightweight col.get(include=[]) to count
matching IDs without fetching documents.
Clean squash by jphein on 2026-04-21. Backwards-compatible via hook_silent_save config flag. Save marker now only advances after confirmed write — strictly safer than status quo.
Merging to fix active pagination crash — 3 users hit this on v3.3.2 in the last 20 hours (54K, 114K, 465K drawer palaces). None-guard regression fixed. 9-day production soak on 165K-drawer palace confirms stability.
Adds a `hook_silent_save` mode (default `true` in new installs) where
the stop and precompact hooks write diary entries directly via the
Python API — no AI block, no MCP tool roundtrip, no possibility of the
AI forgetting or ignoring the save instruction.
**Two modes, controlled by `hook_silent_save` in `~/.mempalace/config.json`:**
1. **Silent mode** (default): Direct call to `tool_diary_write()`. Plain
text, no AI involved, deterministic. Save marker advances only after
the write is confirmed, so mid-save failures do not lose exchanges.
Shows `"✦ N memories woven into the palace"` as a systemMessage
notification so the user knows the save fired.
2. **Block mode** (legacy): Returns `{"decision": "block"}` asking the
AI to call the MCP tool chain. Non-deterministic — the AI may ignore,
summarize lossy, or fail. Kept for backward compatibility.
**Extras rolled in:**
- Block reasons name "MemPalace" explicitly and instruct the AI not to
write to Claude Code's native auto-memory (.md files) — prevents the
two memory systems from stepping on each other.
- Codex transcript handling (`event_msg` payloads) in
`_count_human_messages` + `_extract_recent_messages`.
- Tightened stopword leak in diary summaries; docstring polish; test
hermeticity fixes (per-test `STATE_DIR` patching).
**Tests:** hooks_cli tests cover silent-save path, save-marker
advancement after confirmed write only, and systemMessage formatting.
Rebased fresh on upstream/develop. Only touches files germane to the
feature (hooks_cli.py, tests, hooks/README.md, HOOKS_TUTORIAL.md) —
stale fork-local `.sh` wrapper and plugin manifest changes dropped.
The legacy hook scripts `hooks/mempal_save_hook.sh` and
`hooks/mempal_precompact_hook.sh` shell out to `python3` for JSON
parsing and transcript-message counting. On macOS GUI launches of
Claude Code — `open -a`, Spotlight, the dock — the harness inherits
`PATH` from launchd (`/usr/bin:/bin:/usr/sbin:/sbin`), which may not
contain a `python3` at all, or may contain only a system Python that
lacks what the hook needs. The hook then fails silently in the
background log where users never look.
`mempalace` auto-ingest itself is unaffected — #340 switched that
path to the `mempalace` CLI entry point, which pipx/uv install on a
stable global PATH.
This PR adds a `MEMPAL_PYTHON` environment variable that users can
set to point the hook at any Python 3 interpreter. Resolution order
applied at each `python3` invocation site inside the two hooks:
1. $MEMPAL_PYTHON (if set and executable)
2. $(command -v python3) on PATH
3. bare `python3` as a last resort
The interpreter does not need `mempalace` installed in it — only the
standard-library `json` and `sys` modules. The hook's `mempalace mine`
call runs via the CLI, independent of this override.
hooks/README.md documents the macOS GUI PATH issue and the
MEMPAL_PYTHON override. tests/test_hooks_shell.py adds 3 regression
tests (Linux/macOS only, POSIX bash):
- MEMPAL_PYTHON override wins over PATH (proved via a
marker-emitting shim that proxies to the real interpreter).
- Non-executable MEMPAL_PYTHON falls back to PATH rather than
crashing on permission denied.
- Unset MEMPAL_PYTHON resolves via PATH.
`hooks_cli.py` (the Python implementation invoked via
`mempalace hook run ...`) already uses `sys.executable` and is
therefore trivially correct — no changes needed there.
Supersedes abandoned branch `fix/hook-bugs`.
Co-Authored-By: MSL <232237854+milla-jovovich@users.noreply.github.com>
Three assertions in test_mcp_command_* were still checking for the old
`python -m mempalace.mcp_server` output string. Update to match the new
`mempalace-mcp` command printed by cmd_mcp().
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
cmd_mcp() in cli.py was still printing `python -m mempalace.mcp_server`
as the setup command. Update to use the mempalace-mcp console entry
point added in the previous commit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
hooks/mempal_precompact_hook.sh and hooks/mempal_save_hook.sh used
python3 -m mempalace mine which fails when mempalace is installed via
pipx or uv. Switch to the mempalace CLI entry point which pipx/uv put
on PATH. Also removes the now-unused PYTHON variable from mempal_save_hook.sh.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Hook scripts used `python3 -m mempalace` which fails when mempalace is
installed via pipx or uv. Using the `mempalace` CLI command directly
works for all installation methods. Dev users running from source should
use `pip install -e .` as documented in CONTRIBUTING.md.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The MCP server config used `python -m mempalace.mcp_server` which fails
when mempalace is installed via pipx or uv, since the system python
cannot find the module in the isolated venv. Adding a `mempalace-mcp`
console_scripts entry point ensures the MCP server works regardless of
installation method (pip, pipx, uv, conda).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Addresses bensig's review on PR #1021.
silent_guard was initialized to False, so when both MempalaceConfig
import and .hook_silent_save attribute access failed, silent_guard
stayed False. Then `if not silent_guard:` fired and returned empty —
silently dropping the save. In silent mode (the default since v3.3.0),
saves should ALWAYS proceed on config-read failure. Changing the
initial value to True makes that the safe default.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Upstream develop commit feba7e8 (2026-04-18) added `m = m or {}` to the
single-shot `for m in metas:` loop after this branch already rewrote
status() to paginate. Without porting the guard forward, merging this PR
would silently drop jp's fix and crash again on palaces with null-metadata
drawers.
Addresses bensig's review on #851.
Real bug surfaced on CI for this PR. On POSIX, os.kill(pid, 0) is
the canonical no-op existence probe. On Windows, Python's os.kill
maps to TerminateProcess(handle, sig), which *terminates* the target
with exit code sig. os.kill(pid, 0) therefore kills the target with
exit code 0 — silently destroying our mine child (or, as happened
in test_mine_already_running_live_pid, the pytest process itself).
Fix: split into _pid_alive(pid) helper with a Windows branch using
ctypes.windll.kernel32.OpenProcess + GetExitCodeProcess.
PROCESS_QUERY_LIMITED_INFORMATION opens a handle only if the PID
exists; STILL_ACTIVE (259) distinguishes running from exited processes.
No new dependencies — stdlib ctypes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
On Windows, os.kill(bogus_pid, 0) raises OSError[WinError 87]
"The parameter is incorrect" — NOT ProcessLookupError. The old
except tuple missed it, so test_mine_already_running_dead_pid
failed on Windows CI.
Catching OSError covers ProcessLookupError + PermissionError +
FileNotFoundError on POSIX and WinError 87 on Windows. ValueError
still guards the int() parse.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Every stop hook fire spawned a new background `mempalace mine` via
subprocess.Popen with no dedup — 4 concurrent mines at ~770% CPU
observed in production. Add `_mine_already_running()` (reads
`hook_state/mine.pid`, uses `os.kill(pid, 0)` as an existence check)
and `_spawn_mine()` (writes the child PID to the lock file after
Popen returns). `_maybe_auto_ingest` bails early when the guard
reports True.
Tests: 4 new unit tests for `_mine_already_running` (no file, dead
PID, live PID using `os.getpid()`, corrupt file), 1 new test
covering the skip-when-running branch of `_maybe_auto_ingest`, and
existing spawn tests patched to redirect `_MINE_PID_FILE` into
tmp_path so they don't touch the real state dir.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- _output(): use sys.modules.get() instead of unconditional import to
avoid triggering mcp_server's stdout redirect as a side effect
- _output(): write-all loop for os.write() to handle partial writes and
EINTR; fall back to sys.stdout.buffer on OSError
- _output() docstring: remove inaccurate _save_diary_direct reference
- stop_hook_active guard: narrow except to ImportError/AttributeError,
default silent_guard=False (safe: preserves block-mode loop prevention
when config load fails) and log a warning instead of silently changing
behavior
- tests: two new regression tests covering the real-stdout-fd path and
the fd-1 fallback path
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
mempalace.mcp_server redirects stdout → stderr at module-level import
(both Python-level and fd-level via os.dup2) to protect the MCP stdio
protocol from ChromaDB's C-level noise. Silent-save imports mcp_server
transitively via _save_diary_direct, so by the time _output() calls
print(), sys.stdout is actually stderr.
Claude Code reads hook output from fd 1. With the redirect in effect,
fd 1 points to fd 2, so our {"systemMessage": "✦ N memories woven..."}
JSON lands on stderr and Claude Code never renders it. The save still
happens, the marker still advances — the user just never sees the
beautiful checkpoint notification in their terminal.
Fix: _output() now writes to _REAL_STDOUT_FD (saved by mcp_server before
the redirect) via os.write(), falling back to sys.stdout only when the
saved fd is unavailable (e.g., hooks_cli imported without mcp_server).
Test: bash hook script 2>/dev/null now shows only the JSON;
2>&1 >/dev/null shows only the Diary entry log line — clean separation
restored.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Claude Code 2.1.114 passes stop_hook_active:true on every Stop fire
after the first in a session (plugin-dispatched hooks in particular).
The legacy guard at line 426 was written for block-mode, where a
re-fire with the flag set meant "you already blocked, don't block
again" — correct loop prevention when the hook returns
{"decision":"block"}.
Silent-save mode (default since #673) never blocks — it saves
directly and returns. The flag is meaningless there, so the old
guard was suppressing every auto-save after the first one in a
Claude Code session. Symptom: terminal never shows the "✦ N
memories woven" notification again, hook.log stays silent, save
marker stuck.
Fix: only skip on stop_hook_active when block mode is configured.
Silent mode runs through as normal — the save is deterministic and
idempotent, no loop risk.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a helper that renames HNSW segment directories whose
`data_level0.bin` is significantly older than `chroma.sqlite3`. Drift
between the on-disk HNSW graph and the live embeddings table is the
root cause of a segfault class where the Rust graph-walk dereferences
dangling neighbor pointers for entries in the metadata segment that no
longer exist in the HNSW index, crashing in a background thread on
`count()` or `query()`.
Issue #823 describes the same drift as a silent-staleness symptom
(semantic search returns stale results after `add_drawer` because
`data_level0.bin` lags the sqlite metadata under the default
`sync_threshold=1000`). Under heavier load or after an interrupted
write, the same drift can escalate from "silent stale results" to
"SIGSEGV on next open," which is the failure mode observed at
neo-cortex-mcp#2 (chromadb 1.5.5, Python 3.12) and acknowledged at
chroma-core/chroma#2594.
On one 135K-drawer palace where `index_metadata.pickle` claimed 137,813
elements against 135,464 rows in sqlite (2,349-entry drift), fresh
Python processes crashed in `col.count()` 17/20 times; after renaming
the segment dir out of the way and letting ChromaDB rebuild lazily, the
same 20-run check went to 0 crashes.
The recovery path #823 suggests (export / recreate / reimport) is heavy
— it re-embeds every drawer. This helper is lighter: rename the segment
dir so ChromaDB reopens without it, and the indexer rebuilds lazily on
the next write. The original directory is renamed (not deleted) so the
operator can recover if the heuristic misfires.
If `chroma.sqlite3` is more than `stale_seconds` (default 3600) newer
than the segment's `data_level0.bin`, the segment is considered
suspect. One hour is deliberately conservative — normal HNSW flush
cadence is seconds to minutes, so an hour of drift implies a crashed
mid-write, not routine lag.
- Additive: exposes `quarantine_stale_hnsw(palace_path, stale_seconds)`
as a helper. Not wired into `_client()` / startup on this PR — the
goal is to land the primitive first so operators and higher layers
can opt in. A follow-up could call it automatically on palace open
behind an env var or config flag.
- Closes#823 by giving operators a first-class recovery path without
having to install `chromadb-ops` or re-mine.
Four new tests in `tests/test_backends.py`:
- renames drifted segment, preserves original files under `.drift-TS` suffix
- leaves fresh segments alone
- no-op on missing palace path / missing `chroma.sqlite3`
- skips already-quarantined (`.drift-` suffixed) directories
`pytest tests/test_backends.py` → 11 passed. `ruff check` / `ruff format
--check` — clean.
Same class of bug as #1007: ChromaDB's query() can return None in the
documents and metadatas arrays when a drawer's HNSW vector entry exists
but its metadata/document rows haven't been materialized. The code in
Layer3.search_raw (mempalace/layers.py) calls meta.get("wing", ...),
meta.get("room", ...), meta.get("source_file", ...) directly without
null safety, so it raises:
AttributeError: 'NoneType' object has no attribute 'get'
Two-line defensive coercion matching the pattern in #1009 /
PR #999 for searcher.py: meta = meta or {}, doc = doc or "".
The hit still appears with its real distance; source/wing/room
fall back to their fallback values where the metadata row is missing.
Frequently hit on chromadb 1.5.x (root cause #1006). Even after the
chromadb floor lands (#1010), partial-state results remain possible
during interrupted mines and schema upgrade boundaries, so the guard
is worth having on its own.
Fixes#1011.
The CLI help text and README told first-time users to mine from ~/chats/,
a path that doesn't exist on any machine. Real location where Claude
Code writes session JSONL is ~/.claude/projects/<escaped-project-path>/.
Updates three user-visible strings:
- mempalace/cli.py line 7 ("Two ways to ingest" block)
- mempalace/cli.py line 25 (Examples block)
- README.md line 58 (Quickstart)
Website guides (website/guide/mining.md, getting-started.md) still
reference ~/chats/ for ChatGPT/Slack export scenarios where that remains
a valid placeholder. Those can be a separate PR if the maintainers want
to tilt the website examples toward Claude Code specifically.
Fixes#996.
Five findings from the automated review, fixed with targeted tests where
behavior changed:
1. Transformation Protocol (transforms.py). The registry mixed a bytes-to-str
transform (utf8_replace_invalid) with str-to-str transforms under a single
Callable[..., str] type, misleading static type checkers and adapter
authors. Introduced a Transformation Protocol with __call__(data: bytes|str)
-> str and retyped the registry + get_transformation return.
2. Drawer-id collision risk (context.py). Switched _build_drawer_id from
sha1[:16]=64 bits to sha256[:24]=96 bits. 64 bits sits uncomfortably
close to the birthday bound for palace-sized corpora; 96 bits keeps the
collision probability negligible while preserving the existing
<prefix>_<chunk> layout adapters rely on.
3. Fresh-schema KG columns (knowledge_graph.py). source_drawer_id and
adapter_name now live in the canonical CREATE TABLE so new palaces don't
take an ALTER round-trip on first open. _migrate_schema stays for legacy
palaces (SQLite has no ADD COLUMN IF NOT EXISTS, so PRAGMA introspection
is still needed there).
4. Identity-shim comment (transforms.py). Comment said the adapter-specific
transforms "raise if invoked without adapter context" but they return
the input unchanged. Updated the comment to match the actual identity-
shim behavior Copilot suggested.
5. Test docstring (test_sources.py). Comment mentioned default_factory=list
but SourceRef.options uses default_factory=dict. Corrected.
Tests: 1020 passed (up from 1018), +2 new tests for the sha256 id shape
and the fresh-schema column presence on new palaces.
Four more MCP handlers iterate a metadata list and call m.get(...)
unconditionally. When the cache contains a None entry (drawers with no
metadata, common on older mining paths), the try block catches the
AttributeError and marks the response "partial: true" with an
error message — visible as {"error": "'NoneType' object has no
attribute 'get'", "partial": true} returned from mempalace_status even
though the palace data is otherwise fetchable.
Same m = m or {} guard we applied to searcher.py (d3a2d22, a51c3c2)
and miner.status() (66f08a1). None-metadata drawers now roll up under
the existing "unknown" fallback bucket instead of poisoning the
response with a misleading partial flag.
Regression test: mock the metadata cache with a None in the middle,
assert tool_status returns clean counts and no error/partial fields.
Verified the test fails without the guard.
998 tests pass.
Lands the read-side contract so third-party adapter authors (@Perseusxrltd,
@JakobSachs, @adv3nt3, @zendesk-thittesdorf, @mfhens, @roip, @MrDys) have a
stable target matching what RFC 001 §10 landed on the write side in #995.
Scope (this PR):
- mempalace/sources/base.py: BaseSourceAdapter ABC with kwargs-only
ingest() / describe_schema() and default is_current() / source_summary()
/ close() (§1.1–1.2). Typed records: SourceRef, SourceItemMetadata,
DrawerRecord, RouteHint, SourceSummary, AdapterSchema, FieldSpec (§1.3,
§5.2). Error classes: SourceNotFoundError, AuthRequiredError,
AdapterClosedError, TransformationViolationError, SchemaConformanceError
(§2.7). Class-level identity contract: name / adapter_version /
capabilities / supported_modes / declared_transformations /
default_privacy_class (§2.1, §1.4, §1.5, §6).
- mempalace/sources/transforms.py: reference implementations of the 13
reserved transformations (§1.4) — utf8_replace_invalid, newline_normalize,
whitespace_trim, whitespace_collapse_internal, line_trim, line_join_spaces,
blank_line_drop — as pure functions, plus identity shims for the six
adapter-specific ones (strip_tool_chrome, tool_result_truncate,
tool_result_omitted, spellcheck_user, synthesized_marker,
speaker_role_assignment) that the conversations adapter will override
when migrated. get_transformation(name) resolves by reserved name.
- mempalace/sources/registry.py: entry-point discovery via
importlib.metadata.entry_points(group="mempalace.sources") + explicit
register()/unregister() surface (§3.1–3.2). resolve_adapter_for_source()
implements the §3.3 priority order; crucially, no auto-detection on the
read side (§3.3 is explicit about that — user intent never inferred from
on-disk artifacts).
- mempalace/sources/context.py: PalaceContext facade (§9) bundling the
drawer/closet collections, knowledge graph, palace path, adapter identity,
and progress hooks core passes into adapter.ingest(). upsert_drawer()
applies the spec-mandated adapter_name/adapter_version stamps from §5.1.
skip_current_item() signals laziness; emit() dispatches to hooks and
swallows hook exceptions.
- mempalace/knowledge_graph.py: add_triple() gains optional source_drawer_id
and adapter_name kwargs (§5.5). Backwards-compatible column migration
auto-adds the new columns on open of a pre-RFC 002 palace (PRAGMA
table_info then ALTER TABLE ADD COLUMN), matching the pattern used for
any new palace-side provenance fields.
- pyproject.toml: mempalace.sources entry-point group declared. Empty on
the first-party side for now — miners migrate in a follow-up; the group
being present means third-party packages can begin registering today.
Out of scope (explicit follow-ups):
- miner.py → mempalace/sources/filesystem.py. Behavior-preserving rename
that also moves READABLE_EXTENSIONS, detect_room(), detect_hall() into
the adapter (§9). Larger refactor; lands separately.
- convo_miner.py + normalize.py → mempalace/sources/conversations.py. The
format-detection if-chain in normalize.py becomes per-format plugins;
declared_transformations enumerates what the current pipeline already
does to source bytes (§1.4 existing-code mapping).
- Closet post-step wired into the conversations adapter (§1.7).
- CLI --source flag + --mode deprecation alias (§3.3).
- MCP mempalace_mine tool source parameter.
- AbstractSourceAdapterContractSuite (§7.1–7.3): byte-preservation round-
trip and declared-transformation round-trip tests.
- Privacy-class floor enforcement (§6.2); depends on #389 for
secrets_possible scanning.
Tests: 1018 passed (up from ~990 on develop), +27 targeted tests covering
the ABC instantiation rules, typed records, all reserved transformations,
the registry register/get/unregister surface, PalaceContext upsert + skip +
emit semantics, and both the new KG provenance kwargs and backwards-
compatible legacy-schema migration.
Refs: #989 (RFC 002 tracking), #990 (RFC 002 spec), #995 (RFC 001 §10
cleanup — sibling PR on the write side).
Sathvik-1007