jason/mempalace
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

cleanup and remote only

Browse Source
This commit is contained in:
jason
2026-05-09 10:52:25 -05:00
parent 2fc47a52fc
commit 40e5e5e3cc
136 changed files with 1502 additions and 349529 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hooks/README.md
+91 -83
View File
@@ -2,17 +2,51 @@
These hook scripts make MemPalace save automatically. No manual "save" commands needed.
This deployment ships only the **remote** hook variants — the palace runs as a Docker container on a server (e.g. Unraid), and hooks `curl` the active session transcript to the server's `/ingest/transcript` endpoint over HTTPS with bearer auth. Server-side, the existing `mine_convos` pipeline handles entity detection, room assignment, dedup, and idempotency. See [`deploy/unraid/README.md`](../deploy/unraid/README.md) for the server side.
## What They Do
| Hook | When It Fires | What Happens |
|------|--------------|-------------|
| **Save Hook** | Every 15 human messages | Auto-mines transcript (tool output included), then blocks the AI to save topics/decisions/quotes |
| **PreCompact Hook** | Right before context compaction | Auto-mines transcript, then emergency save — forces the AI to save EVERYTHING before losing context |
|---|---|---|
| **Save Hook** (`mempal_save_hook_remote.sh`) | Every 15 user messages (configurable via `SAVE_INTERVAL`) | Backgrounded `curl` POSTs the active transcript. Returns immediately so the AI doesn't stall. Idempotent — failed retries are safe. |
| **PreCompact Hook** (`mempal_precompact_hook_remote.sh`) | Right before context compaction | Synchronous `curl` POST. Blocks until the upload completes (or the hook timeout fires) so memory is durable before context shrinks. |
**Two-layer capture:** Hooks auto-mine the JSONL transcript directly into the palace (capturing raw tool output — Bash results, search findings, build errors). They also block the AI with a reason message telling it to save verbatim tool output and key context. Belt and suspenders — tool output gets stored even if the AI summarizes instead of quoting.
**Two-layer capture.** The save hook ships the JSONL transcript directly to the server (capturing raw tool output — Bash results, search findings, build errors), where the miner files it verbatim into the palace. Tool output gets stored even if the AI summarizes instead of quoting.
## Env-var contract
The scripts read all configuration from environment variables. There is no script-level config to edit; the same script works against any number of machines.
| Variable | Required | Purpose |
|---|---|---|
| `MEMPAL_REMOTE_URL` | yes | Base URL of the MemPalace server, e.g. `https://unraid.local:8443`. |
| `MEMPAL_REMOTE_TOKEN` | yes | Bearer token shared with the server's `MEMPAL_TOKEN`. |
| `MEMPAL_REMOTE_INSECURE` | no | Set to `1` to skip TLS verification. Use only when the server uses Caddy's `tls internal` self-signed cert and the client hasn't trusted the root CA. |
| `MEMPAL_REMOTE_WING` | no | Force a specific wing for this client's transcripts. Default: server derives wing from the session id. |
| `SAVE_INTERVAL` | no | Override the default of 15 user messages. |
| `MEMPAL_PYTHON` | no | Path to a Python 3 interpreter. Only needs `json` + `sys` from stdlib — mempalace does not need to be installed in it. Used to parse the hook's stdin JSON. |
Set these persistently:
**PowerShell (Windows):**
```powershell
[Environment]::SetEnvironmentVariable("MEMPAL_REMOTE_URL", "https://unraid.local:8443", "User")
[Environment]::SetEnvironmentVariable("MEMPAL_REMOTE_TOKEN", "<the-token>", "User")
[Environment]::SetEnvironmentVariable("MEMPAL_REMOTE_INSECURE", "1", "User") # if self-signed
```
**Bash/Zsh:** add the same exports to `~/.zshrc` / `~/.bashrc`.
If `MEMPAL_REMOTE_URL` or `MEMPAL_REMOTE_TOKEN` is unset, the scripts no-op and log a one-liner — they never block the AI from stopping. Safe to install on a machine that doesn't have a remote configured yet.
## Install — Claude Code
Make the scripts executable:
```bash
chmod +x hooks/mempal_save_hook_remote.sh hooks/mempal_precompact_hook_remote.sh
```
Add to `.claude/settings.local.json`:
```json
@@ -22,26 +56,21 @@ Add to `.claude/settings.local.json`:
"matcher": "*",
"hooks": [{
"type": "command",
"command": "/absolute/path/to/hooks/mempal_save_hook.sh",
"command": "/absolute/path/to/hooks/mempal_save_hook_remote.sh",
"timeout": 30
}]
}],
"PreCompact": [{
"hooks": [{
"type": "command",
"command": "/absolute/path/to/hooks/mempal_precompact_hook.sh",
"timeout": 30
"command": "/absolute/path/to/hooks/mempal_precompact_hook_remote.sh",
"timeout": 60
}]
}]
}
}
```
Make them executable:
```bash
chmod +x hooks/mempal_save_hook.sh hooks/mempal_precompact_hook.sh
```
## Install — Codex CLI (OpenAI)
Add to `.codex/hooks.json`:
@@ -50,132 +79,111 @@ Add to `.codex/hooks.json`:
{
"Stop": [{
"type": "command",
"command": "/absolute/path/to/hooks/mempal_save_hook.sh",
"command": "/absolute/path/to/hooks/mempal_save_hook_remote.sh",
"timeout": 30
}],
"PreCompact": [{
"type": "command",
"command": "/absolute/path/to/hooks/mempal_precompact_hook.sh",
"timeout": 30
"command": "/absolute/path/to/hooks/mempal_precompact_hook_remote.sh",
"timeout": 60
}]
}
```
## Configuration
Edit `mempal_save_hook.sh` to change:
- **`SAVE_INTERVAL=15`** — How many human messages between saves. Lower = more frequent saves, higher = less interruption.
- **`STATE_DIR`** — Where hook state is stored (defaults to `~/.mempalace/hook_state/`)
- **`MEMPAL_DIR`** — Optional **project directory** (code, notes, docs) to also mine on each save trigger, with `--mode projects`. The hook ALWAYS mines the active conversation transcript automatically with `--mode convos` — `MEMPAL_DIR` is purely additive, never an override. Leave blank if you don't want to ingest project files.
- **`MEMPALACE_PYTHON`** — Optional env var. Python interpreter with mempalace + chromadb installed. Auto-detects: `MEMPALACE_PYTHON` env var → repo `venv/bin/python3` → system `python3`. Set this if your venv is in a non-standard location.
### mempalace CLI
The relevant commands are:
```bash
mempalace mine <dir> # Mine all files in a directory
mempalace mine <dir> --mode convos # Mine conversation transcripts only
```
The hooks resolve the repo root automatically from their own path, so they work regardless of where you install the repo.
## How It Works (Technical)
## How it works
### Save Hook (Stop event)
```
User sends message → AI responds → Claude Code fires Stop hook
↓
Hook counts human messages in JSONL transcript
Hook counts user messages in JSONL transcript
↓
┌─── < 15 since last save ──→ echo "{}" (let AI stop)
┌─── < SAVE_INTERVAL since last save ──→ echo "{}" (let AI stop)
│
└─── ≥ 15 since last save
└─── ≥ SAVE_INTERVAL since last save
↓
Auto-mine transcript → palace (tool output captured)
Background curl POST → server /ingest/transcript
↓
{"decision": "block", "reason": "save tool output verbatim..."}
Hook returns {} immediately (AI stops normally)
↓
AI saves to palace (topics, decisions, quotes)
↓
AI tries to stop again
↓
stop_hook_active = true
↓
Hook sees flag → echo "{}" (let it through)
Server-side miner runs in background, files drawers
```
The `stop_hook_active` flag prevents infinite loops: block once → AI saves → tries to stop → flag is true → we let it through.
### PreCompact Hook
```
Context window getting full → Claude Code fires PreCompact
↓
Find transcript (from input or session_id lookup)
Synchronous curl POST → server /ingest/transcript
↓
Auto-mine transcript → palace (tool output captured)
Wait for 200 OK (or hook timeout)
↓
{"decision": "block", "reason": "save tool output verbatim..."}
↓
AI saves everything
↓
Compaction proceeds
echo "{}" → Compaction proceeds
```
No counting needed — compaction always warrants a save. The auto-mine captures raw tool output before the AI gets a chance to summarize it away.
Synchronous on PreCompact is intentional — this is the safety net before context shrinks. The Claude Code hook timeout (set in `settings.local.json`) bounds how long we'll wait.
## Debugging
Check the hook log:
```bash
cat ~/.mempalace/hook_state/hook.log
tail -f ~/.mempalace/hook_state/hook.log
```
Example output:
Example:
```
[14:30:15] Session abc123: 12 exchanges, 12 since last save
[14:35:22] Session abc123: 15 exchanges, 15 since last save
[14:35:22] TRIGGERING SAVE at exchange 15
[14:40:01] Session abc123: 18 exchanges, 3 since last save
[14:35:22] ingest ok
[14:50:18] PRE-COMPACT triggered for session abc123
[14:50:19] PRE-COMPACT ingest ok
```
## Known Limitations
A 401 response means the bearer token is wrong. A connection error means the URL/cert is wrong (or the server is down). All curl output goes to the same log.
**Hooks require session restart after install.** Claude Code loads hooks from `settings.json` at session start only. If you run `mempalace init` or manually edit hook config mid-session, the hooks won't fire until you restart Claude Code. This is a Claude Code limitation.
## Known limitations
**`MEMPAL_PYTHON` override for the hook's internal Python calls.** The save hook parses its JSON input and counts transcript messages with `python3`. When the harness is launched from a GUI on macOS — `open -a`, Spotlight, the dock — its `PATH` is the minimal `/usr/bin:/bin:/usr/sbin:/sbin` inherited from `launchd`, not your shell PATH. If `python3` isn't on that PATH, those internal calls fail and the hook can't count exchanges.
**Hooks require session restart after install.** Claude Code loads hooks from `settings.json` at session start only. If you edit hook config mid-session, restart Claude Code to pick up changes.
Point the hook at any Python 3 interpreter to fix it:
**Python interpreter resolution.** The scripts parse hook stdin JSON with `python3`. When Claude Code is launched from a GUI on macOS (Spotlight, dock, `open -a`), its `PATH` is the minimal `/usr/bin:/bin:/usr/sbin:/sbin` inherited from `launchd` rather than your shell PATH. If `python3` isn't there, set `MEMPAL_PYTHON` to a known-good interpreter:
```bash
export MEMPAL_PYTHON="/usr/bin/python3" # system Python is fine
export MEMPAL_PYTHON="$HOME/.venvs/mempalace/bin/python" # or your venv
export MEMPAL_PYTHON="/usr/bin/python3"
# or:
export MEMPAL_PYTHON="$HOME/.venvs/x/bin/python"
```
Resolution priority: `$MEMPAL_PYTHON` (if set and executable) → `$(command -v python3)` → bare `python3`. The interpreter only needs `json` and `sys` from the standard library — `mempalace` itself does not need to be installed in it.
Resolution priority: `$MEMPAL_PYTHON` → `$(command -v python3)` → bare `python3`. The interpreter only needs `json` and `sys` — mempalace itself does not need to be installed.
Note: the `mempalace mine` auto-ingest runs via the `mempalace` CLI, so that command also needs to be on the hook's `PATH`. Installing with `pipx install mempalace` or `uv tool install mempalace` puts it on a stable global location; otherwise extend the hook environment's `PATH` to include your venv's `bin/`.
**`MineAlreadyRunning` collisions.** If two clients ingest simultaneously, the second one's request returns 500 because the server-side `mine_lock` is held. The save hook is idempotent — the next save catches up. If you see this constantly in the log, raise `SAVE_INTERVAL` on the chattier client.
## Backfill Past Conversations
## Backfilling past conversations
The hooks only capture conversations going forward. To mine **past** Claude Code sessions into your palace, run a one-time backfill:
The hooks only capture sessions going forward. To mine **past** sessions into the remote palace, loop `curl` over them:
```bash
mempalace mine ~/.claude/projects/ --mode convos
# Claude Code sessions
for f in ~/.claude/projects/**/*.jsonl; do
curl -k -X POST \
-H "Authorization: Bearer $MEMPAL_REMOTE_TOKEN" \
-H "X-Session-Id: $(basename "$f" .jsonl)" \
--data-binary @"$f" \
"$MEMPAL_REMOTE_URL/ingest/transcript"
done
# Codex CLI sessions
for f in ~/.codex/sessions/**/*.jsonl; do
curl -k -X POST \
-H "Authorization: Bearer $MEMPAL_REMOTE_TOKEN" \
-H "X-Session-Id: $(basename "$f" .jsonl)" \
--data-binary @"$f" \
"$MEMPAL_REMOTE_URL/ingest/transcript"
done
```
This scans all JSONL transcripts from previous sessions and files them into the `conversations` wing. On a typical developer machine with months of history, this can yield 50K–200K drawers.
For Codex CLI sessions:
```bash
mempalace mine ~/.codex/sessions/ --mode convos
```
This only needs to be done once — after that, the hooks auto-mine each session as you go.
The server-side miner is idempotent — re-uploading the same transcript won't double-file. Drop `-k` once Caddy's root CA is trusted on the client.
## Cost
**Zero extra tokens.** The hooks notify the AI that saves happened in the background — the AI doesn't need to write anything in the chat. All filing is handled automatically. Previous versions asked the AI to write diary entries and drawer content in the chat window, which cost ~$1/session in retransmitted tokens.
**Zero extra tokens.** The hooks save in the background — the AI doesn't need to write anything in the chat window. All filing happens server-side after the upload returns.