Igor Lins e Silva
fe56797762
Address Copilot review on #1231: 1. Stop double-mining the transcript on the Python side. ``_get_mine_targets`` now returns only the ``MEMPAL_DIR`` projects target — the convos target for the transcript dir is dropped because ``_ingest_transcript`` already handles it on every hook fire. The duplicate spawn was using ``sys.executable`` (vs ``_mempalace_python()``) and a different ``--wing``, so each Stop/PreCompact event was writing the same transcript into two wings under asymmetric interpreters and overwriting the single ``_MINE_PID_FILE`` lock. 2. ``_maybe_auto_ingest`` and ``_mine_sync`` now spawn via ``_mempalace_python()`` so the resolved interpreter matches the venv that owns mempalace (matters under GUI-launched harnesses where ``sys.executable`` may resolve to a system Python without chromadb). 3. Replace ``eval $(...)`` in both shell hooks with a ``mapfile``-based reader. Sanitized values are still emitted by the same Python parser, but the shell now does plain variable assignment instead of executing the parser's stdout — smaller blast radius if the sanitizer is ever bypassed. 4. Mirror ``_validate_transcript_path`` in the shell hooks via a ``is_valid_transcript_path`` helper — extension + traversal-segment rejection, parity with the Python validator. The convos mine in each shell hook is now gated on the validator instead of bare ``-f``. 5. Tighten the ``..`` traversal test that previously exercised the suffix gate by mistake (``../../etc/passwd`` lacks ``.json[l]``). Use ``.jsonl`` paths with traversal segments to actually hit the ``..`` rejection branch. 6. README: add a one-liner pointing at ``mempalace sweep`` for users who want per-message recall on top of the file-level chunks the hooks produce. The sweeper was undiscoverable previously. Tests: 1418 passed, 1 skipped (full suite minus benchmarks).
MemPalace Hooks — Auto-Save for Terminal AI Tools
These hook scripts make MemPalace save automatically. No manual "save" commands needed.
What They Do
| Hook | When It Fires | What Happens |
|---|---|---|
| Save Hook | Every 15 human messages | Auto-mines transcript (tool output included), then blocks the AI to save topics/decisions/quotes |
| PreCompact Hook | Right before context compaction | Auto-mines transcript, then emergency save — forces the AI to save EVERYTHING before losing context |
Two-layer capture: Hooks auto-mine the JSONL transcript directly into the palace (capturing raw tool output — Bash results, search findings, build errors). They also block the AI with a reason message telling it to save verbatim tool output and key context. Belt and suspenders — tool output gets stored even if the AI summarizes instead of quoting.
Install — Claude Code
Add to .claude/settings.local.json:
{
"hooks": {
"Stop": [{
"matcher": "*",
"hooks": [{
"type": "command",
"command": "/absolute/path/to/hooks/mempal_save_hook.sh",
"timeout": 30
}]
}],
"PreCompact": [{
"hooks": [{
"type": "command",
"command": "/absolute/path/to/hooks/mempal_precompact_hook.sh",
"timeout": 30
}]
}]
}
}
Make them executable:
chmod +x hooks/mempal_save_hook.sh hooks/mempal_precompact_hook.sh
Install — Codex CLI (OpenAI)
Add to .codex/hooks.json:
{
"Stop": [{
"type": "command",
"command": "/absolute/path/to/hooks/mempal_save_hook.sh",
"timeout": 30
}],
"PreCompact": [{
"type": "command",
"command": "/absolute/path/to/hooks/mempal_precompact_hook.sh",
"timeout": 30
}]
}
Configuration
Edit mempal_save_hook.sh to change:
SAVE_INTERVAL=15— How many human messages between saves. Lower = more frequent saves, higher = less interruption.STATE_DIR— Where hook state is stored (defaults to~/.mempalace/hook_state/)MEMPAL_DIR— Optional project directory (code, notes, docs) to also mine on each save trigger, with--mode projects. The hook ALWAYS mines the active conversation transcript automatically with--mode convos—MEMPAL_DIRis purely additive, never an override. Leave blank if you don't want to ingest project files.MEMPALACE_PYTHON— Optional env var. Python interpreter with mempalace + chromadb installed. Auto-detects:MEMPALACE_PYTHONenv var → repovenv/bin/python3→ systempython3. Set this if your venv is in a non-standard location.
mempalace CLI
The relevant commands are:
mempalace mine <dir> # Mine all files in a directory
mempalace mine <dir> --mode convos # Mine conversation transcripts only
The hooks resolve the repo root automatically from their own path, so they work regardless of where you install the repo.
How It Works (Technical)
Save Hook (Stop event)
User sends message → AI responds → Claude Code fires Stop hook
↓
Hook counts human messages in JSONL transcript
↓
┌─── < 15 since last save ──→ echo "{}" (let AI stop)
│
└─── ≥ 15 since last save
↓
Auto-mine transcript → palace (tool output captured)
↓
{"decision": "block", "reason": "save tool output verbatim..."}
↓
AI saves to palace (topics, decisions, quotes)
↓
AI tries to stop again
↓
stop_hook_active = true
↓
Hook sees flag → echo "{}" (let it through)
The stop_hook_active flag prevents infinite loops: block once → AI saves → tries to stop → flag is true → we let it through.
PreCompact Hook
Context window getting full → Claude Code fires PreCompact
↓
Find transcript (from input or session_id lookup)
↓
Auto-mine transcript → palace (tool output captured)
↓
{"decision": "block", "reason": "save tool output verbatim..."}
↓
AI saves everything
↓
Compaction proceeds
No counting needed — compaction always warrants a save. The auto-mine captures raw tool output before the AI gets a chance to summarize it away.
Debugging
Check the hook log:
cat ~/.mempalace/hook_state/hook.log
Example output:
[14:30:15] Session abc123: 12 exchanges, 12 since last save
[14:35:22] Session abc123: 15 exchanges, 15 since last save
[14:35:22] TRIGGERING SAVE at exchange 15
[14:40:01] Session abc123: 18 exchanges, 3 since last save
Known Limitations
Hooks require session restart after install. Claude Code loads hooks from settings.json at session start only. If you run mempalace init or manually edit hook config mid-session, the hooks won't fire until you restart Claude Code. This is a Claude Code limitation.
MEMPAL_PYTHON override for the hook's internal Python calls. The save hook parses its JSON input and counts transcript messages with python3. When the harness is launched from a GUI on macOS — open -a, Spotlight, the dock — its PATH is the minimal /usr/bin:/bin:/usr/sbin:/sbin inherited from launchd, not your shell PATH. If python3 isn't on that PATH, those internal calls fail and the hook can't count exchanges.
Point the hook at any Python 3 interpreter to fix it:
export MEMPAL_PYTHON="/usr/bin/python3" # system Python is fine
export MEMPAL_PYTHON="$HOME/.venvs/mempalace/bin/python" # or your venv
Resolution priority: $MEMPAL_PYTHON (if set and executable) → $(command -v python3) → bare python3. The interpreter only needs json and sys from the standard library — mempalace itself does not need to be installed in it.
Note: the mempalace mine auto-ingest runs via the mempalace CLI, so that command also needs to be on the hook's PATH. Installing with pipx install mempalace or uv tool install mempalace puts it on a stable global location; otherwise extend the hook environment's PATH to include your venv's bin/.
Backfill Past Conversations
The hooks only capture conversations going forward. To mine past Claude Code sessions into your palace, run a one-time backfill:
mempalace mine ~/.claude/projects/ --mode convos
This scans all JSONL transcripts from previous sessions and files them into the conversations wing. On a typical developer machine with months of history, this can yield 50K–200K drawers.
For Codex CLI sessions:
mempalace mine ~/.codex/sessions/ --mode convos
This only needs to be done once — after that, the hooks auto-mine each session as you go.
Cost
Zero extra tokens. The hooks notify the AI that saves happened in the background — the AI doesn't need to write anything in the chat. All filing is handled automatically. Previous versions asked the AI to write diary entries and drawer content in the chat window, which cost ~$1/session in retransmitted tokens.