cleanup and remote only

This commit is contained in:
2026-05-09 10:52:25 -05:00
parent 2fc47a52fc
commit 40e5e5e3cc
136 changed files with 1502 additions and 349529 deletions
+61
View File
@@ -0,0 +1,61 @@
# MemPalace Caddy reverse-proxy config.
# -----------------------------------------------------------------------------
# Listens on :8443 with a self-signed (Caddy-internal) cert. Enforces a
# bearer-token check on every request and proxies authenticated traffic to
# the mempalace container.
#
# Two upstream paths:
# /sse, /messages* -> mempalace:8765 (mcp-proxy SSE for MCP tool calls)
# /ingest* -> mempalace:8766 (in-process HTTP ingest endpoint)
# /healthz -> mempalace:8766 (no auth, for liveness probes)
#
# Token comes from the MEMPAL_TOKEN env var (set in deploy/unraid/.env).
# -----------------------------------------------------------------------------
{
# Disable the admin API — never expose it from a container that's
# reachable from clients.
admin off
# Ship access logs to stderr so `docker logs caddy` is useful.
log {
output stderr
format console
}
}
:8443 {
tls internal
# Liveness probe — no auth so Docker / external monitors can hit it
# without holding the bearer token.
handle /healthz {
reverse_proxy mempalace:8766
}
# Auth gate. matcher passes only when the Authorization header is
# exactly `Bearer ${MEMPAL_TOKEN}`. Header matching is exact-match.
@authorized header Authorization "Bearer {$MEMPAL_TOKEN}"
# MCP-over-SSE: the MCP transport sends events on /sse and accepts
# JSON-RPC POSTs on /messages (path varies by mcp-proxy version, so
# proxy the whole prefix tree).
handle @authorized {
# SSE responses are streamed — disable buffering and force HTTP/1.1
# upstream to keep the event stream open.
reverse_proxy /sse* /messages* mempalace:8765 {
flush_interval -1
transport http {
versions 1.1
}
}
reverse_proxy /ingest* mempalace:8766
}
# Default: anything not matched above (or unauthenticated traffic) is
# rejected. Returning 401 instead of 403 is correct here — clients with
# no/invalid token can re-attempt with credentials.
respond 401 {
body "Unauthorized"
close
}
}
+512
View File
@@ -0,0 +1,512 @@
# MemPalace on Unraid — server-mode deployment
This directory contains everything needed to run MemPalace as a shared
memory server on an Unraid box and connect multiple AI tools (Claude
Code, Codex, Antigravity, or any MCP-compatible client) to a single
persistent palace.
If you only use one machine, you don't need any of this — install
mempalace locally per the main [README](../../README.md) and you're
done. This guide is for users running the same AI tools across multiple
machines who want one shared memory.
---
## What you get
```
home LAN
┌───────────────────────────────────┐
│ Unraid (always on) │
│ ┌────────────────────────────┐ │
│ │ caddy :8443 (TLS + auth) │ │
│ │ ├─ /sse → mcp-proxy │ │
│ │ └─ /ingest → ingest API │ │
│ │ mempalace (single process) │ │
│ │ ├─ mcp-proxy :8765 │ │
│ │ └─ ingest :8766 │ │
│ └────────────────────────────┘ │
│ /mnt/user/appdata/mempalace/ │
│ ├─ palace/ ChromaDB │
│ ├─ kg/ knowledge graph │
│ └─ inbox/ uploaded sessions │
└───────────────────────────────────┘
│ │ │
┌─────┴─┐ ┌────┴──┐ ┌───┴──────┐
│ box A │ │ box B │ │ box C │
│ Claude│ │ Codex │ │ Antigrav │
└───────┘ └───────┘ └──────────┘
```
* **One palace, many clients.** Search and write target the same
ChromaDB index regardless of which machine you're on.
* **Auto-save hooks work across machines.** Each client's session
transcripts get pushed to the server on `Stop` and `PreCompact`
events; the server-side miner runs the existing `mine_convos`
pipeline (entity detection, room assignment, dedup, idempotency).
* **Single shared secret.** One bearer token gates both MCP and
transcript ingest at the Caddy edge.
What this is **not**: a multi-tenant cloud product. There's one palace,
one token, no per-user isolation. It's designed for a single user with
multiple machines.
---
## Files in this directory
| File | Purpose |
|---|---|
| `docker-compose.yml` | Two-container stack: `mempalace` + `caddy` sidecar. |
| `Caddyfile` | Caddy config: bearer-token auth, self-signed TLS, SSE-aware reverse proxy. |
| `mempalace-server.xml` | dockerMan template for a single-container, **no-auth, LAN-trust-only** install (compose path is the recommended one). |
| `README.md` | This file. |
The `Dockerfile` and `.dockerignore` live at the repo root — the compose
build context is `../..` so it can reach them.
---
## Prerequisites
* Unraid 6.12+ with Docker enabled (default).
* The **Compose Manager** plugin from Community Apps. Required for the
recommended (auth-enabled) path. The dockerMan template path doesn't
need it but has no auth.
* `/mnt/user/appdata` set up (default on every Unraid).
* Ports `8443` free on the Unraid host (or change in `docker-compose.yml`).
You do **not** need Tailscale, WireGuard, a domain name, a public IP,
SWAG, or NPM. The stack is self-contained.
---
## Install (recommended: compose with auth)
### 1. Get the repo onto Unraid
SSH to Unraid, pick a path on a regular share (not `/boot`, not
`/mnt/cache` directly), and clone or copy the repo:
```bash
mkdir -p /mnt/user/system/build
cd /mnt/user/system/build
git clone <your-fork-or-rsync-source> mempalace
cd mempalace/deploy/unraid
```
### 2. Mint a bearer token
```bash
TOKEN=$(openssl rand -hex 32)
echo "MEMPAL_TOKEN=$TOKEN" > .env
chmod 600 .env
echo "Token: $TOKEN" # save to a password manager — you'll set this on each client
```
`MEMPAL_TOKEN` is read from `.env` by `docker compose`. The same token
is forwarded to:
* Caddy, which checks `Authorization: Bearer <token>` on every request.
* The in-container ingest server as `MEMPALACE_INGEST_TOKEN` for
defense-in-depth.
### 3. Create the appdata directories
```bash
mkdir -p /mnt/user/appdata/mempalace \
/mnt/user/appdata/mempalace-caddy/data \
/mnt/user/appdata/mempalace-caddy/config
chown -R 99:100 /mnt/user/appdata/mempalace
chown -R 99:100 /mnt/user/appdata/mempalace-caddy
```
The Caddy data dir holds Caddy's auto-generated root CA — back it up
so re-deploys keep the same cert (clients won't have to re-trust it).
### 4. Build and start
```bash
docker compose up -d --build
```
First build downloads Python 3.13-slim and pip-installs `mempalace` +
`mcp-proxy` (~35 min on a Celeron, faster on real hardware).
### 5. Verify
```bash
# unauth'd liveness probe
curl -k https://<unraid-ip>:8443/healthz
# → {"status":"ok","version":"3.3.x"}
# bearer-checked endpoint should 401 without the token
curl -ki https://<unraid-ip>:8443/ingest/transcript
# HTTP/2 401
# ...and accept a request with it
curl -k -H "Authorization: Bearer $TOKEN" https://<unraid-ip>:8443/healthz
# → 200 OK
```
If you see all of the above, the server is up and the auth gate is
working.
### 6. (Optional) Trust Caddy's root CA on each client
Caddy's `tls internal` directive auto-generates a self-signed root CA
on first start. Clients must either trust that CA or skip TLS
verification (`-k` for curl, `MEMPAL_REMOTE_INSECURE=1` for hooks,
disabled SSL verify for the MCP client).
To trust it once and stop seeing TLS warnings:
```bash
# On Unraid:
cat /mnt/user/appdata/mempalace-caddy/data/caddy/pki/authorities/local/root.crt
```
Copy that PEM block to each Windows client and import into the
**Trusted Root Certification Authorities** store via `certmgr.msc`,
or via PowerShell:
```powershell
Import-Certificate -FilePath C:\path\to\root.crt -CertStoreLocation Cert:\LocalMachine\Root
```
---
## Connect AI tools
You'll need [`mcp-proxy`](https://github.com/sparfenyuk/mcp-proxy) on
each client machine:
```bash
uv tool install mcp-proxy
# or:
pip install mcp-proxy
```
Set environment variables persistently. **PowerShell** (Windows):
```powershell
[Environment]::SetEnvironmentVariable("MEMPAL_REMOTE_URL", "https://<unraid-ip>:8443", "User")
[Environment]::SetEnvironmentVariable("MEMPAL_REMOTE_TOKEN", "<the-token>", "User")
# Drop this once you've trusted Caddy's root CA:
[Environment]::SetEnvironmentVariable("MEMPAL_REMOTE_INSECURE", "1", "User")
```
**Bash/Zsh** (macOS/Linux): add the same three exports to
`~/.zshrc` / `~/.bashrc`.
### Claude Code
Add to `~/.claude.json` (user-scoped) or `.mcp.json` in the project:
```json
{
"mcpServers": {
"mempalace": {
"command": "mcp-proxy",
"args": [
"https://<unraid-ip>:8443/sse",
"--headers", "Authorization", "Bearer <the-token>"
],
"env": {
"PYTHONHTTPSVERIFY": "0"
}
}
}
}
```
Drop the `env` block once Caddy's root CA is trusted on the client.
### Codex CLI
Add to `~/.codex/config.toml`:
```toml
[mcp_servers.mempalace]
command = "mcp-proxy"
args = [
"https://<unraid-ip>:8443/sse",
"--headers", "Authorization", "Bearer <the-token>",
]
[mcp_servers.mempalace.env]
PYTHONHTTPSVERIFY = "0"
```
### Antigravity
Antigravity uses the Windsurf-derived MCP layout. Open the IDE's
MCP settings UI (Settings → AI → MCP Servers) and add:
```json
{
"mempalace": {
"command": "mcp-proxy",
"args": [
"https://<unraid-ip>:8443/sse",
"--headers", "Authorization", "Bearer <the-token>"
]
}
}
```
Or edit `~/.antigravity/mcp.json` directly with the same shape.
### Verify each client
In any of the three tools, start a session and call:
> "Use mempalace_status to show palace stats."
Expected: a JSON blob with `total_drawers`, wing/room breakdown, etc.
A 401 means the token is wrong; a connection error means the
URL/cert is wrong.
---
## Set up auto-save hooks
The `_remote.sh` hook variants in `../../hooks/` push transcripts to
the server instead of running `mempalace mine` locally. They share the
same env-var contract as the MCP client config above.
### Claude Code
Make the scripts executable:
```bash
chmod +x hooks/mempal_save_hook_remote.sh \
hooks/mempal_precompact_hook_remote.sh
```
Add to `.claude/settings.local.json`:
```json
{
"hooks": {
"Stop": [{
"matcher": "*",
"hooks": [{
"type": "command",
"command": "/abs/path/to/hooks/mempal_save_hook_remote.sh",
"timeout": 30
}]
}],
"PreCompact": [{
"hooks": [{
"type": "command",
"command": "/abs/path/to/hooks/mempal_precompact_hook_remote.sh",
"timeout": 60
}]
}]
}
}
```
### Codex CLI
Add to `.codex/hooks.json` with the same shape — the scripts are
hook-host-agnostic.
### What the hooks do
| Hook | Trigger | Behavior |
|---|---|---|
| `mempal_save_hook_remote.sh` | Every 15 user messages (configurable via `SAVE_INTERVAL` env var) | Backgrounded `curl` POSTs the active transcript to `/ingest/transcript`. Returns immediately so the AI doesn't stall. Idempotent — failed retries are safe. |
| `mempal_precompact_hook_remote.sh` | Right before context compaction | Synchronous `curl` POST. Blocks until the upload completes (or the hook timeout fires) so memory is durable before context shrinks. |
Both write logs to `~/.mempalace/hook_state/hook.log`. Tail it during
setup to confirm uploads are landing.
### Optional env vars
| Variable | Default | Purpose |
|---|---|---|
| `MEMPAL_REMOTE_URL` | *(required)* | Server base URL, e.g. `https://unraid.local:8443`. |
| `MEMPAL_REMOTE_TOKEN` | *(required)* | Bearer token. |
| `MEMPAL_REMOTE_INSECURE` | unset | Set to `1` to skip TLS verification. Use only with `tls internal`. |
| `MEMPAL_REMOTE_WING` | unset | Force a specific wing for this client's transcripts. Default: server derives wing from session id. |
| `SAVE_INTERVAL` | `15` | Messages between save-hook fires. |
---
## Backfilling history
The hooks only capture sessions going forward. To mine **past**
transcripts into the remote palace, on each client run:
```bash
curl -k -X POST \
-H "Authorization: Bearer $MEMPAL_REMOTE_TOKEN" \
-H "X-Session-Id: backfill-$(hostname)-$(date +%s)" \
-H "X-Wing: backfill" \
--data-binary @/path/to/some-session.jsonl \
"$MEMPAL_REMOTE_URL/ingest/transcript"
```
For a whole directory of past sessions, loop:
```bash
for f in ~/.claude/projects/**/*.jsonl; do
curl -k -X POST \
-H "Authorization: Bearer $MEMPAL_REMOTE_TOKEN" \
-H "X-Session-Id: $(basename "$f" .jsonl)" \
--data-binary @"$f" \
"$MEMPAL_REMOTE_URL/ingest/transcript"
done
```
The server-side miner is idempotent — re-uploading the same transcript
won't double-file.
---
## Backups
Everything that matters lives in `/mnt/user/appdata/mempalace/`:
* `palace/` — ChromaDB vector index + SQLite metadata
* `kg/` — knowledge-graph SQLite
* `inbox/` — uploaded transcripts (kept for re-mining if needed)
Add it to your **CA Backup / Appdata Backup** schedule. Losing this
directory loses all memory.
The Caddy data dir (`/mnt/user/appdata/mempalace-caddy/data/`) is also
worth backing up — it contains the auto-generated root CA. Without it,
re-deploys regenerate the CA and clients have to re-trust it.
---
## dockerMan template (no-auth, LAN-trust-only)
If you don't want auth and trust your LAN absolutely (no other people,
no untrusted IoT, no guests), the `mempalace-server.xml` template gives
you a single-container, dockerMan-compatible install:
```bash
# Build the image:
cd /mnt/user/system/build/mempalace
docker build -t mempalace-server:latest .
# Install the template:
cp deploy/unraid/mempalace-server.xml \
/boot/config/plugins/dockerMan/templates-user/my-MemPalace.xml
```
Then in the Unraid WebUI: Docker → Add Container → "Select a template" →
**MemPalace** → Apply.
This path skips Caddy entirely. The MCP SSE endpoint is published bare
on `:8765`, no TLS, no auth. Anyone on the LAN can read and write the
palace. **Only use this if you understand and accept that.**
---
## Troubleshooting
### `mcp-proxy` connects but tool calls hang
Caddy is buffering SSE responses. Verify `flush_interval -1` is set in
the Caddyfile and that Caddy version is 2.7+ (the compose pulls
`caddy:2-alpine` which is current).
### 401 from every request
The token in the client's MCP config doesn't match the server's
`MEMPAL_TOKEN`. Print both to confirm:
```bash
# On Unraid:
grep MEMPAL_TOKEN /mnt/user/system/build/mempalace/deploy/unraid/.env
# On client (PowerShell):
[Environment]::GetEnvironmentVariable("MEMPAL_REMOTE_TOKEN", "User")
```
### `MineAlreadyRunning` errors in hook logs
Two clients hit the ingest endpoint simultaneously. The server-side
miner serializes via `mine_lock` and rejects the second one. The hook
is idempotent — the next save catches up. If you see this constantly,
raise `SAVE_INTERVAL` on the chattier client.
### Caddy logs `tls: handshake failure`
Client doesn't trust the self-signed cert. Either trust the root CA
(see step 6 in install) or set `MEMPAL_REMOTE_INSECURE=1` /
`PYTHONHTTPSVERIFY=0` on that client.
### Container can't start: "address already in use"
Port 8443 is taken (commonly by Unraid's WebUI HTTPS or another
service). Edit `docker-compose.yml` and change the host-side mapping:
```yaml
ports:
- "9443:8443" # change 9443 to whatever's free
```
Update `MEMPAL_REMOTE_URL` on every client to match.
### Embedding model download stalls on first request
The ~80 MB MiniLM ONNX model downloads from HuggingFace on first
use. Slow connections can time out the initial mining call. Pre-warm
it manually:
```bash
docker exec mempalace python -c \
"from chromadb.utils.embedding_functions import ONNXMiniLM_L6_V2; ONNXMiniLM_L6_V2()(['warmup'])"
```
Subsequent uses load from `/data/.cache/chroma/` — ~50 ms.
### Logs
```bash
docker logs mempalace # MCP server, ingest server
docker logs mempalace-caddy # auth gate, TLS, access logs
tail -f ~/.mempalace/hook_state/hook.log # client-side hook activity
```
---
## Updating
When this repo updates upstream:
```bash
cd /mnt/user/system/build/mempalace
git pull
cd deploy/unraid
docker compose up -d --build
```
Compose only rebuilds the `mempalace` service (the image hash
changes); Caddy is pinned to `caddy:2-alpine` and pulls latest within
the 2.x line.
Your palace data and Caddy CA persist across rebuilds because they're
on volumes outside the container.
---
## Going further
* **Replace self-signed TLS with Let's Encrypt** — point a real domain at
Unraid (DDNS or otherwise), open port 80 for ACME challenge, and
change `tls internal` in `Caddyfile` to `tls your@email`. Caddy
handles the rest.
* **Put behind SWAG / Nginx Proxy Manager** — drop the Caddy sidecar,
keep `mempalace` exposing 8765/8766 internally only, and add the
routes to your existing reverse proxy. Bearer-token auth and SSE
pass-through must be configured manually.
* **Per-machine wings** — set `MEMPAL_REMOTE_WING=<machinename>` on
each client so transcripts file under separate wings; cross-wing
search still works via the palace graph.
+82
View File
@@ -0,0 +1,82 @@
# MemPalace Unraid Compose
# -----------------------------------------------------------------------------
# Two-container stack: mempalace (MCP-over-SSE on 8765 + HTTP ingest on 8766,
# both bound to localhost only) plus a Caddy sidecar that terminates TLS,
# enforces a bearer token, and reverse-proxies both endpoints on :8443.
#
# Use this with the Unraid Compose Manager plugin. Build context is the
# repo root (../..); on Unraid, sync the repo to /mnt/user/<somewhere>/mempalace
# and from this directory run:
#
# # 1. Generate a token (do this once, keep it secret):
# openssl rand -hex 32 > .env.token
# echo "MEMPAL_TOKEN=$(cat .env.token)" > .env
# rm .env.token
#
# # 2. Build and start:
# docker compose up -d --build
#
# Endpoints (after start):
# https://<unraid-ip>:8443/sse — MCP for AI clients
# https://<unraid-ip>:8443/ingest/... — transcript uploads from hooks
# https://<unraid-ip>:8443/healthz — liveness, no auth
#
# Caddy uses a self-signed cert (`tls internal`); clients must accept it,
# typically via a `--insecure`-style flag or by trusting the Caddy root CA.
# -----------------------------------------------------------------------------
services:
mempalace:
build:
context: ../..
dockerfile: Dockerfile
image: mempalace-server:latest
container_name: mempalace
restart: unless-stopped
# Not published on the host — only Caddy reaches these ports over the
# internal compose network. This is the auth boundary.
expose:
- "8765"
- "8766"
volumes:
- /mnt/user/appdata/mempalace:/data
environment:
MEMPALACE_PALACE_PATH: /data/palace
MEMPALACE_INGEST_PORT: "8766"
MEMPALACE_INGEST_HOST: "0.0.0.0"
# Defense-in-depth — Caddy is the primary gate, but if it's bypassed
# (e.g. someone exec'd into the container's network), the ingest
# server still requires the token.
MEMPALACE_INGEST_TOKEN: "${MEMPAL_TOKEN}"
# Languages for entity detection (comma-separated):
# MEMPALACE_ENTITY_LANGUAGES: en
user: "99:100"
networks:
- mempal
# Override the image CMD: bind mcp-proxy to all interfaces inside the
# container network so Caddy can reach it. The ingest server thread
# spawns from MEMPALACE_INGEST_PORT.
command: >
mcp-proxy --sse-host 0.0.0.0 --sse-port 8765
--pass-environment -- mempalace-mcp
caddy:
image: caddy:2-alpine
container_name: mempalace-caddy
restart: unless-stopped
depends_on:
- mempalace
ports:
- "8443:8443"
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile:ro
- /mnt/user/appdata/mempalace-caddy/data:/data
- /mnt/user/appdata/mempalace-caddy/config:/config
environment:
MEMPAL_TOKEN: "${MEMPAL_TOKEN}"
networks:
- mempal
networks:
mempal:
driver: bridge
+99
View File
@@ -0,0 +1,99 @@
<?xml version="1.0"?>
<Container version="2">
<Name>MemPalace</Name>
<Repository>mempalace-server:latest</Repository>
<Registry>https://github.com/MemPalace/mempalace</Registry>
<Network>bridge</Network>
<MyIP/>
<Shell>sh</Shell>
<Privileged>false</Privileged>
<Support>https://github.com/MemPalace/mempalace/issues</Support>
<Project>https://github.com/MemPalace/mempalace</Project>
<Overview>
Local-first AI memory server. Stores conversations and project content
verbatim in a searchable palace, exposed to MCP-compatible AI tools
(Claude Code, Codex, Antigravity, etc.) over Server-Sent Events on
port 8765.
The image is built locally — see Dockerfile in the repo root. From the
Unraid CLI:
cd /mnt/user/&lt;path&gt;/mempalace
docker build -t mempalace-server:latest .
Then add this template via Add Container -- Template.
Mount /mnt/user/appdata/mempalace to /data for persistent storage of
the ChromaDB index, SQLite knowledge graph, and embedding-model cache.
SECURITY: this container exposes the MCP endpoint without authentication.
Bind it to a trusted network (LAN-only or Tailscale) or place it behind
SWAG / Nginx Proxy Manager with bearer-token or basic auth.
Endpoint: http://[UNRAID-IP]:8765/sse
</Overview>
<Category>Productivity: Tools: Other:</Category>
<WebUI/>
<TemplateURL/>
<Icon>https://raw.githubusercontent.com/MemPalace/mempalace/develop/assets/mempalace_logo.png</Icon>
<ExtraParams>--user 99:100</ExtraParams>
<PostArgs/>
<CPUset/>
<DateInstalled/>
<DonateText/>
<DonateLink/>
<Description>
Persistent AI memory across machines. Connect Claude Code, Codex,
Antigravity, or any MCP-compatible client to a single shared palace.
</Description>
<Config Name="MCP SSE port"
Target="8765"
Default="8765"
Mode="tcp"
Description="Port the MCP-over-SSE endpoint listens on. Clients connect to http://[UNRAID-IP]:[PORT]/sse."
Type="Port"
Display="always"
Required="true"
Mask="false">8765</Config>
<Config Name="Appdata"
Target="/data"
Default="/mnt/user/appdata/mempalace"
Mode="rw"
Description="Persistent storage for the palace (ChromaDB), knowledge graph (SQLite), embedding-model cache, and config."
Type="Path"
Display="always"
Required="true"
Mask="false">/mnt/user/appdata/mempalace</Config>
<Config Name="Palace path (inside container)"
Target="MEMPALACE_PALACE_PATH"
Default="/data/palace"
Mode=""
Description="Subdirectory inside /data where ChromaDB files live. Change only if migrating from a different layout."
Type="Variable"
Display="advanced"
Required="false"
Mask="false">/data/palace</Config>
<Config Name="Embedding device"
Target="MEMPALACE_EMBEDDING_DEVICE"
Default=""
Mode=""
Description="ONNX execution provider: cpu | cuda | dml | coreml. Leave blank for auto. CUDA requires the NVIDIA Driver plugin and GPU passthrough; the image must be rebuilt with the [gpu] extra installed."
Type="Variable"
Display="advanced"
Required="false"
Mask="false"></Config>
<Config Name="Entity-detection languages"
Target="MEMPALACE_ENTITY_LANGUAGES"
Default="en"
Mode=""
Description="Comma-separated language codes for entity detection (e.g. en,es,de)."
Type="Variable"
Display="advanced"
Required="false"
Mask="false">en</Config>
</Container>