feat: deterministic hook saves — zero data loss via silent Python API
Adds a `hook_silent_save` mode (default `true` in new installs) where
the stop and precompact hooks write diary entries directly via the
Python API — no AI block, no MCP tool roundtrip, no possibility of the
AI forgetting or ignoring the save instruction.
**Two modes, controlled by `hook_silent_save` in `~/.mempalace/config.json`:**
1. **Silent mode** (default): Direct call to `tool_diary_write()`. Plain
text, no AI involved, deterministic. Save marker advances only after
the write is confirmed, so mid-save failures do not lose exchanges.
Shows `"✦ N memories woven into the palace"` as a systemMessage
notification so the user knows the save fired.
2. **Block mode** (legacy): Returns `{"decision": "block"}` asking the
AI to call the MCP tool chain. Non-deterministic — the AI may ignore,
summarize lossy, or fail. Kept for backward compatibility.
**Extras rolled in:**
- Block reasons name "MemPalace" explicitly and instruct the AI not to
write to Claude Code's native auto-memory (.md files) — prevents the
two memory systems from stepping on each other.
- Codex transcript handling (`event_msg` payloads) in
`_count_human_messages` + `_extract_recent_messages`.
- Tightened stopword leak in diary summaries; docstring polish; test
hermeticity fixes (per-test `STATE_DIR` patching).
**Tests:** hooks_cli tests cover silent-save path, save-marker
advancement after confirmed write only, and systemMessage formatting.
Rebased fresh on upstream/develop. Only touches files germane to the
feature (hooks_cli.py, tests, hooks/README.md, HOOKS_TUTORIAL.md) —
stale fork-local `.sh` wrapper and plugin manifest changes dropped.
This commit is contained in:
jp
+40
-14
@@ -6,10 +6,10 @@ These hook scripts make MemPalace save automatically. No manual "save" commands
|
||||
|
||||
| Hook | When It Fires | What Happens |
|
||||
|------|--------------|-------------|
|
||||
| **Save Hook** | Every 15 human messages | Blocks the AI, tells it to save key topics/decisions/quotes to the palace |
|
||||
| **PreCompact Hook** | Right before context compaction | Emergency save — forces the AI to save EVERYTHING before losing context |
|
||||
| **Save Hook** | Every 15 human messages | Auto-mines transcript (tool output included), then blocks the AI to save topics/decisions/quotes |
|
||||
| **PreCompact Hook** | Right before context compaction | Auto-mines transcript, then emergency save — forces the AI to save EVERYTHING before losing context |
|
||||
|
||||
The AI does the actual filing — it knows the conversation context, so it classifies memories into the right wings/halls/closets. The hooks just tell it WHEN to save.
|
||||
**Two-layer capture:** Hooks auto-mine the JSONL transcript directly into the palace (capturing raw tool output — Bash results, search findings, build errors). They also block the AI with a reason message telling it to save verbatim tool output and key context. Belt and suspenders — tool output gets stored even if the AI summarizes instead of quoting.
|
||||
|
||||
## Install — Claude Code
|
||||
|
||||
@@ -68,6 +68,7 @@ Edit `mempal_save_hook.sh` to change:
|
||||
- **`SAVE_INTERVAL=15`** — How many human messages between saves. Lower = more frequent saves, higher = less interruption.
|
||||
- **`STATE_DIR`** — Where hook state is stored (defaults to `~/.mempalace/hook_state/`)
|
||||
- **`MEMPAL_DIR`** — Optional. Set to a conversations directory to auto-run `mempalace mine <dir>` on each save trigger. Leave blank (default) to let the AI handle saving via the block reason message.
|
||||
- **`MEMPALACE_PYTHON`** — Optional env var. Python interpreter with mempalace + chromadb installed. Auto-detects: `MEMPALACE_PYTHON` env var → repo `venv/bin/python3` → system `python3`. Set this if your venv is in a non-standard location.
|
||||
|
||||
### mempalace CLI
|
||||
|
||||
@@ -91,15 +92,19 @@ User sends message → AI responds → Claude Code fires Stop hook
|
||||
↓
|
||||
┌─── < 15 since last save ──→ echo "{}" (let AI stop)
|
||||
│
|
||||
└─── ≥ 15 since last save ──→ {"decision": "block", "reason": "save..."}
|
||||
↓
|
||||
AI saves to palace
|
||||
↓
|
||||
AI tries to stop again
|
||||
↓
|
||||
stop_hook_active = true
|
||||
↓
|
||||
Hook sees flag → echo "{}" (let it through)
|
||||
└─── ≥ 15 since last save
|
||||
↓
|
||||
Auto-mine transcript → palace (tool output captured)
|
||||
↓
|
||||
{"decision": "block", "reason": "save tool output verbatim..."}
|
||||
↓
|
||||
AI saves to palace (topics, decisions, quotes)
|
||||
↓
|
||||
AI tries to stop again
|
||||
↓
|
||||
stop_hook_active = true
|
||||
↓
|
||||
Hook sees flag → echo "{}" (let it through)
|
||||
```
|
||||
|
||||
The `stop_hook_active` flag prevents infinite loops: block once → AI saves → tries to stop → flag is true → we let it through.
|
||||
@@ -109,14 +114,18 @@ The `stop_hook_active` flag prevents infinite loops: block once → AI saves →
|
||||
```
|
||||
Context window getting full → Claude Code fires PreCompact
|
||||
↓
|
||||
Hook ALWAYS blocks
|
||||
Find transcript (from input or session_id lookup)
|
||||
↓
|
||||
Auto-mine transcript → palace (tool output captured)
|
||||
↓
|
||||
{"decision": "block", "reason": "save tool output verbatim..."}
|
||||
↓
|
||||
AI saves everything
|
||||
↓
|
||||
Compaction proceeds
|
||||
```
|
||||
|
||||
No counting needed — compaction always warrants a save.
|
||||
No counting needed — compaction always warrants a save. The auto-mine captures raw tool output before the AI gets a chance to summarize it away.
|
||||
|
||||
## Debugging
|
||||
|
||||
@@ -150,6 +159,23 @@ Resolution priority: `$MEMPAL_PYTHON` (if set and executable) → `$(command -v
|
||||
|
||||
Note: the `mempalace mine` auto-ingest runs via the `mempalace` CLI, so that command also needs to be on the hook's `PATH`. Installing with `pipx install mempalace` or `uv tool install mempalace` puts it on a stable global location; otherwise extend the hook environment's `PATH` to include your venv's `bin/`.
|
||||
|
||||
## Backfill Past Conversations
|
||||
|
||||
The hooks only capture conversations going forward. To mine **past** Claude Code sessions into your palace, run a one-time backfill:
|
||||
|
||||
```bash
|
||||
mempalace mine ~/.claude/projects/ --mode convos
|
||||
```
|
||||
|
||||
This scans all JSONL transcripts from previous sessions and files them into the `conversations` wing. On a typical developer machine with months of history, this can yield 50K–200K drawers.
|
||||
|
||||
For Codex CLI sessions:
|
||||
```bash
|
||||
mempalace mine ~/.codex/sessions/ --mode convos
|
||||
```
|
||||
|
||||
This only needs to be done once — after that, the hooks auto-mine each session as you go.
|
||||
|
||||
## Cost
|
||||
|
||||
**Zero extra tokens.** The hooks notify the AI that saves happened in the background — the AI doesn't need to write anything in the chat. All filing is handled automatically. Previous versions asked the AI to write diary entries and drawer content in the chat window, which cost ~$1/session in retransmitted tokens.
|
||||
|
||||
Reference in New Issue
Block a user