ver-0.7
This commit is contained in:
Jason Stedwell
BIN
Binary file not shown.
@@ -30,6 +30,49 @@ The endpoint has a **valid TLS certificate**, so `-k` is not needed (add it only
|
||||
- Complete endpoint→logic routing map (every write destination, its trigger, and why it's distinct): `references/routing-map.md`
|
||||
- Full API reference with every endpoint pattern and the memory routing map: `references/api-reference.md`
|
||||
|
||||
Executable logic ships under `scripts/`:
|
||||
|
||||
- `scripts/echo.sh` — the **validated API client**; prefer it over hand-built `curl` (below)
|
||||
- `scripts/routing.json` — the **canonical, machine-readable** route manifest (the routing map's source of truth; the linter enforces it)
|
||||
- `scripts/vault-lint.sh` — read-only invariant checker (Vault Health)
|
||||
- `scripts/bootstrap.sh` / `scripts/migrate.sh` — deterministic vault setup/repair and schema migration
|
||||
|
||||
## Bundled Tooling (prefer over raw curl)
|
||||
|
||||
All paths below are under `${CLAUDE_PLUGIN_ROOT}/skills/echo-memory/`.
|
||||
|
||||
**`scripts/echo.sh` — use this for every read/write.** It centralizes auth, **HTTP-status checking** (a failed write exits non-zero instead of looking like success), one bounded retry on 5xx/connection errors, idempotent append, correct `::` heading targets, and frontmatter patches. The raw `curl` recipes later in this file are the underlying mechanics / fallback — reach for them only if `echo.sh` is unavailable, and if you do, **check the HTTP status yourself** (the PATCH-heading `400 invalid-target` failure silently loses writes otherwise).
|
||||
|
||||
```bash
|
||||
ECHO="${CLAUDE_PLUGIN_ROOT}/skills/echo-memory/scripts/echo.sh"
|
||||
"$ECHO" get <path> # 404 -> exit 44
|
||||
"$ECHO" ls <dir> ; "$ECHO" map <path> # listing / document-map
|
||||
"$ECHO" search <terms...>
|
||||
"$ECHO" put <path> <file> # create/overwrite (read-back verified)
|
||||
"$ECHO" append <path> "<line>" # idempotent: skips if the exact line exists
|
||||
"$ECHO" patch <path> append heading "<H1::Sub>" <file>
|
||||
"$ECHO" fm <path> updated '"2026-06-19"' ; "$ECHO" bump <path> # frontmatter
|
||||
"$ECHO" lock <session-id> ; "$ECHO" unlock <session-id>
|
||||
```
|
||||
|
||||
**Bootstrap / migrate** are scripts now, not hand-run curl loops: `scripts/bootstrap.sh [--dry-run]` (idempotent, probe-before-write, never overwrites) and `scripts/migrate.sh [--apply]` (reads the marker's `schema_version` and applies migrations; dry-run by default). See `references/bootstrap.md`.
|
||||
|
||||
### Concurrency — the vault is shared, so coordinate writes
|
||||
|
||||
ECHO is read/written by multiple clients (Claude Code **and** CoWork sessions). The single-line files (`heartbeat/last-session.md`, `current-context.md::Scope`, `inbox.md`) assume a single writer at a time. Before a burst of writes in a session that may overlap another, take the **advisory lock**, and release it at session end:
|
||||
|
||||
```bash
|
||||
"$ECHO" lock "cc-$(date +%s)" # exit 75 if another session holds a fresh lock
|
||||
# ... do the writes ...
|
||||
"$ECHO" unlock "cc-$(date +%s)"
|
||||
```
|
||||
|
||||
The lock is cooperative (a stale lock past `ECHO_LOCK_TTL`, default 15 min, is reclaimable) and lives at `_agent/locks/vault.lock`. It is a courtesy, not a hard mutex — if you can't take it, tell Jason another session may be active rather than racing it.
|
||||
|
||||
### Slash commands
|
||||
|
||||
`/echo-load` (cold-start read), `/echo-save <text>` (route + persist), `/echo-triage` (drain the inbox), `/echo-health` (run the linter). These are explicit entry points to the procedures below.
|
||||
|
||||
## Operating Contract & Safety
|
||||
|
||||
The vault is the **system of record** for long-term memory, not a scratchpad. Default to **additive updates, explicit status changes, and traceable summaries**. Keep agent-managed content (`agent_written: true` + `source_notes`) separable from human-authored content. Non-negotiable safety rules:
|
||||
@@ -294,20 +337,22 @@ A weekly/monthly rollup is a **light digest** — open threads across `projects/
|
||||
|
||||
On the first substantive session of a calendar month, run a quick health pass and write findings to `_agent/health/YYYY-MM-vault-health.md`. This is **agent self-maintenance, not a journal entry** — it lives under `_agent/` because it's about the vault's mechanical integrity, not Jason's work narrative. Don't auto-fix without asking.
|
||||
|
||||
Run the bundled linter first — it mechanically checks the invariants below so you don't eyeball them:
|
||||
Run the bundled linter first — it mechanically checks the invariants below so you don't eyeball them. **Pass `ECHO_TODAY` = the conversation's `currentDate`** so stale/aging math uses the same clock you write with (not the runner's machine date):
|
||||
|
||||
```bash
|
||||
bash "${CLAUDE_PLUGIN_ROOT}/skills/echo-memory/scripts/vault-lint.sh"
|
||||
ECHO_TODAY=<currentDate> "${CLAUDE_PLUGIN_ROOT}/skills/echo-memory/scripts/vault-lint.sh"
|
||||
```
|
||||
|
||||
Checks (the linter asserts each and prints violations):
|
||||
Exit codes: `0` clean · `1` violations · `2` unreachable · `3` not bootstrapped. Checks (the linter asserts each and prints violations):
|
||||
|
||||
1. **Stale active projects** — for each note in `projects/active/`, check `updated:` >30 days. Likely belongs in `on-hold/`.
|
||||
2. **Unprocessed inbox** — GET `inbox/captures/inbox.md`. List items older than 14 days that never moved through the triage protocol.
|
||||
3. **Duplicate slugs across lifecycle folders** — any slug appearing in more than one of `active/`, `incubating/`, `on-hold/`, `archived/` is broken state.
|
||||
4. **Folder ↔ `status:` mismatch** — any `projects/<lifecycle>/` note whose `status:` frontmatter disagrees with its folder.
|
||||
5. **Wikilinks in frontmatter** — any `[[...]]` inside a YAML frontmatter block (breaks Obsidian reading view).
|
||||
5. **Wikilinks in frontmatter** — any `[[...]]` inside a YAML frontmatter block (breaks Obsidian reading view), swept across all folders.
|
||||
6. **Duplicate `## Agent Log` headings** — any daily note with more than one.
|
||||
7. **Unknown / retired paths** — any vault file that matches no route in `scripts/routing.json` (or sits at a retired path).
|
||||
8. **Frontmatter integrity** — missing required fields, `updated` < `created`, future `updated`, and `[[wikilinks]]` leaking into `source_notes`.
|
||||
|
||||
The pass is cheap and pays for itself by catching drift before it requires a reorg. Write the findings as a digest; act on them only with Jason's go-ahead.
|
||||
|
||||
|
||||
@@ -4,6 +4,8 @@ Server: `https://echoapi.alwisp.com` (reverse proxy → backend Obsidian Local R
|
||||
Auth header: `Authorization: Bearer 241265fbe6830934a9a4ad3e69335f64a42153b663aa5b0017cb1ea1217b2bab`
|
||||
The endpoint has a **valid TLS certificate** — `-k` is not required. Paths address the vault at its **root**.
|
||||
|
||||
> **Prefer `scripts/echo.sh` over the raw recipes below.** It wraps every verb with auth, status checking, retry, idempotent append, and frontmatter patches. The recipes here are the underlying mechanics and the fallback. **If you call `curl` directly, check the HTTP status** — add `-o /dev/null -w "%{http_code}"` and branch on it. A `PATCH` to a non-existent heading returns `400 invalid-target` (errorCode 40080) and the write is *silently lost*; a bare `curl` that ignores status will report success anyway. `GET` returns `404` for a missing file. Treat any `>= 400` as a failed operation, surface it, and do not continue as if it succeeded.
|
||||
|
||||
---
|
||||
|
||||
## Reading Files
|
||||
|
||||
@@ -4,6 +4,20 @@ The **plugin is the single source of truth** for ECHO's structure. Everything ne
|
||||
|
||||
The vault holds **data only**. It carries no `CLAUDE.md` / `BOOTSTRAP.md` / `STRUCTURE.md` / `index.md`. The "is this vault set up?" signal is a small marker file, `_agent/echo-vault.md`.
|
||||
|
||||
## Quick path — run the scripts
|
||||
|
||||
Bootstrap, repair, and migration are deterministic scripts; prefer them over running the curl steps by hand. They resolve the scaffold relative to their own location, so they work regardless of the caller's CWD:
|
||||
|
||||
```bash
|
||||
SCRIPTS="${CLAUDE_PLUGIN_ROOT}/skills/echo-memory/scripts"
|
||||
"$SCRIPTS/bootstrap.sh" --dry-run # preview what would be seeded
|
||||
"$SCRIPTS/bootstrap.sh" # idempotent, additive — fills only what is missing (also the repair path)
|
||||
"$SCRIPTS/migrate.sh" # plan a schema migration (dry-run)
|
||||
"$SCRIPTS/migrate.sh" --apply # perform the migration (moves/deletes, after review)
|
||||
```
|
||||
|
||||
`bootstrap.sh` writes through `echo.sh`, so every step is status-checked and the marker is written last. The manual steps below document what the script does (and serve as a fallback if the script can't run).
|
||||
|
||||
```
|
||||
AUTH="Authorization: Bearer 241265fbe6830934a9a4ad3e69335f64a42153b663aa5b0017cb1ea1217b2bab"
|
||||
BASE="https://echoapi.alwisp.com"
|
||||
@@ -75,9 +89,12 @@ PUT every file under this skill's `scaffold/templates/` to its mirrored vault pa
|
||||
|
||||
Templates keep their Obsidian Templater tokens (`{{date:YYYY-MM-DD}}` etc.) verbatim — those are resolved by Templater / by the skill's daily-note routine, not at seed time.
|
||||
|
||||
Resolve scaffold paths against the skill directory — **never a bare relative `@scaffold/...`**, which assumes the caller's CWD is the skill dir and silently sends an empty body otherwise:
|
||||
|
||||
```bash
|
||||
SCAFFOLD="${CLAUDE_PLUGIN_ROOT}/skills/echo-memory/scaffold"
|
||||
curl -s -X PUT -H "$AUTH" -H "Content-Type: text/markdown" \
|
||||
--data-binary @scaffold/templates/journal/templates/daily-note-template.md \
|
||||
--data-binary @"$SCAFFOLD/templates/journal/templates/daily-note-template.md" \
|
||||
"$BASE/vault/journal/templates/daily-note-template.md"
|
||||
```
|
||||
|
||||
@@ -115,6 +132,8 @@ Run the same steps 1–5, but GET-probe each path first and **only create what i
|
||||
|
||||
## Migrations (`schema_version` mismatch)
|
||||
|
||||
`scripts/migrate.sh` automates this: it reads the marker's `schema_version`, applies each intervening migration (idempotent, additive; destructive steps gated behind `--apply` and printed first), and stamps the marker at the end. Run it dry-run, review the plan, then `--apply`. The steps below are what it encodes — and the manual fallback.
|
||||
|
||||
When the marker's `schema_version` is older than the plugin's, apply the migration steps for each intervening version, then PATCH the marker's `schema_version` frontmatter to the new value.
|
||||
|
||||
- **0 → 1** (control-docs-in-plugin): the vault previously carried root control docs (`CLAUDE.md`, `BOOTSTRAP.md`, `STRUCTURE.md`, `index.md`). Back them up outside the vault, DELETE them, PUT the thin `scaffold/README.vault.md` over the old verbose `README.md`, write the `_agent/echo-vault.md` marker, and scrub now-dangling `[[CLAUDE]]`/`[[BOOTSTRAP]]`/`[[STRUCTURE]]`/`[[index]]` links from the `## Related` sections of `operator-preferences.md` and `current-context.md` (leave historical session logs alone). Confirm with the operator before deleting.
|
||||
|
||||
@@ -34,3 +34,12 @@ You are an agent operating against an Obsidian vault that functions as a shared,
|
||||
## REST/API readiness
|
||||
|
||||
Assume clients may operate without filesystem access, through the Obsidian Local REST API. Keep paths predictable, frontmatter parseable, titles stable, and all state stored in notes rather than hidden conversation memory. Structure must stay portable: a fresh, empty vault is brought fully online by the plugin's bootstrap (`references/bootstrap.md`), so nothing essential should depend on files existing in the vault ahead of time.
|
||||
|
||||
## Concurrency (shared vault)
|
||||
|
||||
The vault is a **shared** substrate — Claude Code, CoWork, and other REST/MCP clients may operate on it concurrently. The REST API offers no transactions, so writers coordinate cooperatively:
|
||||
|
||||
- Single-line, overwrite-style files (`_agent/heartbeat/last-session.md`, `current-context.md::Scope`) and append targets (`inbox.md`, `## Agent Log`) assume **one writer at a time**. Two sessions writing at once can clobber or duplicate.
|
||||
- Before a burst of writes in a session that may overlap another, take the advisory lock (`echo.sh lock <id>` → `_agent/locks/vault.lock`) and release it at session end. The lock is cooperative with a TTL (stale locks are reclaimable); it is a courtesy, not a hard mutex.
|
||||
- Idempotent append (read-before-POST, via `echo.sh append`) is the second line of defense against duplicate lines from retries or overlapping sessions.
|
||||
- Status-check every write. A write that returns `>= 400` did **not** land — surface it rather than assuming success.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**This document is canonical and complete.** Every write destination in the vault appears here exactly once, with the condition that routes content to it, what lands there, and why it is distinct from its neighbours. The rule for the whole system: **if a path is not in this map, nothing is written to it.** A path that cannot justify its separateness from a neighbour is a merge candidate, not a valid destination.
|
||||
|
||||
Three views of the same truth: the `SKILL.md` *Where to Write* table is the quick-reference, this map is the authority, and `vault-layout.md` is the physical tree. When they disagree, this map wins and the others are fixed to match.
|
||||
Views of the same truth: `scripts/routing.json` is the **machine-readable canonical source** (one route per destination, as a regex pattern + trigger + reason); `vault-lint.sh` loads it and flags any vault path that matches no route (and any write to a retired path). This prose map is the human-readable authority and must stay in sync with `routing.json`. The `SKILL.md` *Where to Write* table is the quick-reference and `vault-layout.md` is the physical tree. When they disagree, `routing.json` + this map win and the others are fixed to match.
|
||||
|
||||
## How to read a row
|
||||
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
#!/usr/bin/env bash
|
||||
# bootstrap.sh — stand up (or repair) an ECHO vault deterministically.
|
||||
#
|
||||
# Idempotent and additive: every write is probe-before-write and NEVER overwrites an
|
||||
# existing file. The marker (_agent/echo-vault.md) is written LAST, so the vault is
|
||||
# only flagged "set up" once every piece is in place. Safe to re-run any time — that
|
||||
# is also the "repair" path (it fills in only what is missing).
|
||||
#
|
||||
# All scaffold is resolved relative to THIS script's location, so it works regardless
|
||||
# of the caller's CWD (fixes the old `@scaffold/...` relative-path assumption).
|
||||
#
|
||||
# Usage:
|
||||
# bootstrap.sh [--dry-run]
|
||||
#
|
||||
# Env: ECHO_BASE, ECHO_KEY (passed through to echo.sh), ECHO_TODAY (YYYY-MM-DD for {{DATE}}).
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SKILL_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
SCAFFOLD="$SKILL_DIR/scaffold"
|
||||
ECHO="$SCRIPT_DIR/echo.sh"
|
||||
TODAY="${ECHO_TODAY:-$(date +%Y-%m-%d)}"
|
||||
DRY=0
|
||||
[ "${1:-}" = "--dry-run" ] || [ "${1:-}" = "-n" ] && DRY=1
|
||||
|
||||
[ -d "$SCAFFOLD" ] || { echo "bootstrap: scaffold not found at $SCAFFOLD" >&2; exit 1; }
|
||||
[ -x "$ECHO" ] || chmod +x "$ECHO" 2>/dev/null || true
|
||||
|
||||
say() { echo "bootstrap: $*"; }
|
||||
exists() { ECHO_VERIFY=0 "$ECHO" get "$1" >/dev/null 2>&1; } # 0 = present(200), nonzero = absent/404
|
||||
|
||||
# seed VAULT_PATH from LOCAL_FILE (with {{DATE}} substitution), only if absent
|
||||
seed() {
|
||||
local vpath="$1" local_file="$2"
|
||||
if exists "$vpath"; then say "skip (exists) $vpath"; return 0; fi
|
||||
if [ "$DRY" = "1" ]; then say "would seed $vpath <- ${local_file#$SKILL_DIR/}"; return 0; fi
|
||||
sed "s/{{DATE}}/$TODAY/g" "$local_file" | ECHO_VERIFY=1 "$ECHO" put "$vpath" - >/dev/null
|
||||
say "seeded $vpath"
|
||||
}
|
||||
|
||||
# write a one-line leaf README only if absent
|
||||
leaf_readme() {
|
||||
local dir="$1" name="${1##*/}"
|
||||
local vpath="$dir/README.md"
|
||||
if exists "$vpath"; then return 0; fi
|
||||
if [ "$DRY" = "1" ]; then say "would readme $vpath"; return 0; fi
|
||||
printf '# %s\n\nMemory vault folder. See the echo-memory plugin for conventions.\n' "$name" \
|
||||
| ECHO_VERIFY=0 "$ECHO" put "$vpath" - >/dev/null
|
||||
say "readme $vpath"
|
||||
}
|
||||
|
||||
# ---- Pre-flight: is the vault already bootstrapped? --------------------------
|
||||
if exists "_agent/echo-vault.md"; then
|
||||
ver="$("$ECHO" get _agent/echo-vault.md 2>/dev/null | sed -n 's/^schema_version:[[:space:]]*//p' | head -1)"
|
||||
say "marker present (schema_version=${ver:-unknown}). Running repair pass (fills only missing files)."
|
||||
CUR_SCHEMA=2
|
||||
if [ -n "$ver" ] && [ "$ver" -lt "$CUR_SCHEMA" ] 2>/dev/null; then
|
||||
say "NOTE: schema_version $ver < $CUR_SCHEMA — run migrate.sh before relying on the vault."
|
||||
fi
|
||||
fi
|
||||
|
||||
# ---- 1. Folder tree (leaf READMEs guarantee non-empty dirs) ------------------
|
||||
LEAVES=(
|
||||
inbox/captures inbox/imports inbox/processing-log
|
||||
journal/daily journal/weekly journal/monthly journal/quarterly journal/annual journal/templates
|
||||
projects/active projects/incubating projects/on-hold projects/archived
|
||||
areas/business areas/personal areas/learning areas/systems
|
||||
resources/concepts resources/references resources/people resources/companies resources/meetings
|
||||
decisions/by-date
|
||||
_agent/context _agent/memory/working _agent/memory/episodic _agent/memory/semantic
|
||||
_agent/sessions _agent/health _agent/templates _agent/heartbeat
|
||||
_agent/skills/active _agent/skills/archived _agent/locks
|
||||
)
|
||||
for d in "${LEAVES[@]}"; do leaf_readme "$d"; done
|
||||
|
||||
# ---- 2. Templates (mirror scaffold/templates/ 1:1 into the vault) ------------
|
||||
if [ -d "$SCAFFOLD/templates" ]; then
|
||||
while IFS= read -r f; do
|
||||
rel="${f#$SCAFFOLD/templates/}"
|
||||
seed "$rel" "$f"
|
||||
done < <(find "$SCAFFOLD/templates" -type f -name '*.md')
|
||||
fi
|
||||
|
||||
# ---- 3. Anchor seeds (only if absent — never fabricate facts) ----------------
|
||||
seed "_agent/memory/semantic/operator-preferences.md" "$SCAFFOLD/anchors/operator-preferences.seed.md"
|
||||
seed "_agent/context/current-context.md" "$SCAFFOLD/anchors/current-context.seed.md"
|
||||
seed "inbox/captures/inbox.md" "$SCAFFOLD/anchors/inbox.seed.md"
|
||||
|
||||
# ---- 4. Vault README (human signpost) ----------------------------------------
|
||||
seed "README.md" "$SCAFFOLD/README.vault.md"
|
||||
|
||||
# ---- 5. Marker (write LAST) --------------------------------------------------
|
||||
seed "_agent/echo-vault.md" "$SCAFFOLD/echo-vault.md"
|
||||
|
||||
say "done (${DRY:+DRY-RUN }$TODAY)."
|
||||
say "Next: create today's daily note + a bootstrap session log + heartbeat (see SKILL.md First-run trace)."
|
||||
+199
@@ -0,0 +1,199 @@
|
||||
#!/usr/bin/env bash
|
||||
# echo.sh — the single validated client for the ECHO Obsidian Local REST API.
|
||||
#
|
||||
# Every read/write to the vault should go through this script instead of hand-built
|
||||
# curl. It centralizes: auth injection, HTTP-status checking (non-zero exit on >=400
|
||||
# so a failed write can never look like success), one bounded retry on 5xx/connection
|
||||
# errors, idempotent append (read-before-POST), correct `::` heading-target handling,
|
||||
# frontmatter field patches, and an advisory multi-writer lock.
|
||||
#
|
||||
# Config (env overrides; defaults match the rest of the plugin):
|
||||
# ECHO_BASE default https://echoapi.alwisp.com
|
||||
# ECHO_KEY default the plugin bearer token
|
||||
# ECHO_VERIFY default 1 — read-back verify after a PUT
|
||||
# ECHO_LOCK_TTL default 900 — seconds before an advisory lock is considered stale
|
||||
#
|
||||
# Usage:
|
||||
# echo.sh get <path> # print file contents (404 -> exit 44)
|
||||
# echo.sh map <path> # document-map JSON (headings/blocks/frontmatter)
|
||||
# echo.sh ls <dir> # directory listing JSON
|
||||
# echo.sh search <query...> # /search/simple
|
||||
# echo.sh put <path> [file] # create/overwrite (body from file or stdin)
|
||||
# echo.sh post <path> [file] # raw append (NON-idempotent; prefer `append`)
|
||||
# echo.sh append <path> <line> # idempotent append: skips if the exact line exists
|
||||
# echo.sh patch <path> <append|prepend|replace> <heading|frontmatter|block> <target> [file]
|
||||
# echo.sh fm <path> <field> <json-value> # PATCH a frontmatter scalar (e.g. fm p.md updated '"2026-06-19"')
|
||||
# echo.sh bump <path> [YYYY-MM-DD] # set frontmatter updated: to today (or given date)
|
||||
# echo.sh delete <path> # DELETE (destructive; explicit use only)
|
||||
# echo.sh lock <owner-id> # acquire advisory lock (exit 75 if held by someone else & fresh)
|
||||
# echo.sh unlock <owner-id> # release advisory lock if owned by <owner-id>
|
||||
#
|
||||
# Exit codes: 0 ok · 44 not-found(404) · 75 lock-held · 2 usage · 1 other HTTP/transport error.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ECHO_BASE="${ECHO_BASE:-https://echoapi.alwisp.com}"
|
||||
ECHO_BASE="${ECHO_BASE%/}"
|
||||
ECHO_KEY="${ECHO_KEY:-241265fbe6830934a9a4ad3e69335f64a42153b663aa5b0017cb1ea1217b2bab}"
|
||||
ECHO_VERIFY="${ECHO_VERIFY:-1}"
|
||||
ECHO_LOCK_TTL="${ECHO_LOCK_TTL:-900}"
|
||||
AUTH="Authorization: Bearer ${ECHO_KEY}"
|
||||
|
||||
# response-body scratch file (filled by every _curl; read by callers via $RESP)
|
||||
RESP="$(mktemp)"; BODY=""
|
||||
cleanup() { rm -f "$RESP" "${BODY:-}"; }
|
||||
trap cleanup EXIT
|
||||
|
||||
die() { echo "echo.sh: $*" >&2; exit 1; }
|
||||
usage() { sed -n '2,40p' "$0" >&2; exit 2; }
|
||||
|
||||
_today() { echo "${ECHO_TODAY:-$(date +%Y-%m-%d)}"; }
|
||||
_now_iso() { date -u +%Y-%m-%dT%H:%M:%SZ; }
|
||||
_vault_url() { echo "${ECHO_BASE}/vault/$1"; }
|
||||
|
||||
# _curl METHOD URL [extra curl args...]
|
||||
# Writes the response body to $RESP and the status code to $HTTP — BOTH IN THE PARENT
|
||||
# SHELL (never call this in $(...) or on the right of a pipe, or those globals are lost).
|
||||
# One bounded retry on transport failure (000) or 5xx.
|
||||
HTTP=""
|
||||
_curl() {
|
||||
local method="$1" url="$2"; shift 2
|
||||
local code attempt=0
|
||||
while :; do
|
||||
code="$(curl -sS -X "$method" -H "$AUTH" -o "$RESP" -w '%{http_code}' "$@" "$url" 2>/dev/null || echo 000)"
|
||||
if { [ "$code" = "000" ] || [ "${code:0:1}" = "5" ]; } && [ "$attempt" -lt 1 ]; then
|
||||
attempt=$((attempt+1)); sleep 1; continue
|
||||
fi
|
||||
break
|
||||
done
|
||||
HTTP="$code"
|
||||
}
|
||||
|
||||
# assert $HTTP is acceptable; 404 -> exit 44, other >=400 -> exit 1
|
||||
_check() {
|
||||
local ctx="$1"
|
||||
[ "$HTTP" = "404" ] && exit 44
|
||||
[ "$HTTP" = "000" ] && die "$ctx: vault unreachable (connection failed) [$ECHO_BASE]"
|
||||
[ "${HTTP:-000}" -ge 400 ] && die "$ctx: HTTP $HTTP — $(cat "$RESP")"
|
||||
return 0
|
||||
}
|
||||
|
||||
# capture a body argument (file path or '-'/empty for stdin) into $BODY (a temp file)
|
||||
_capture_body() {
|
||||
BODY="$(mktemp)"
|
||||
if [ "${1:-}" = "" ] || [ "${1:-}" = "-" ]; then cat > "$BODY"; else cat "$1" > "$BODY"; fi
|
||||
}
|
||||
|
||||
cmd="${1:-}"; shift || true
|
||||
case "$cmd" in
|
||||
get)
|
||||
[ $# -ge 1 ] || usage
|
||||
_curl GET "$(_vault_url "$1")"; _check "get $1"; cat "$RESP" ;;
|
||||
|
||||
map)
|
||||
[ $# -ge 1 ] || usage
|
||||
_curl GET "$(_vault_url "$1")" -H 'Accept: application/vnd.olrapi.document-map+json'
|
||||
_check "map $1"; cat "$RESP" ;;
|
||||
|
||||
ls)
|
||||
[ $# -ge 1 ] || usage
|
||||
p="$1"; [ "${p%/}" = "$p" ] && p="$p/"
|
||||
_curl GET "$(_vault_url "$p")"; _check "ls $1"; cat "$RESP" ;;
|
||||
|
||||
search)
|
||||
[ $# -ge 1 ] || usage
|
||||
q="$*"; q="${q// /+}"
|
||||
_curl POST "${ECHO_BASE}/search/simple/?query=${q}"; _check "search"; cat "$RESP" ;;
|
||||
|
||||
put)
|
||||
[ $# -ge 1 ] || usage
|
||||
path="$1"; _capture_body "${2:-}"
|
||||
_curl PUT "$(_vault_url "$path")" -H 'Content-Type: text/markdown' --data-binary @"$BODY"
|
||||
_check "put $path"
|
||||
if [ "$ECHO_VERIFY" = "1" ]; then
|
||||
_curl GET "$(_vault_url "$path")"
|
||||
[ "$HTTP" = "200" ] || die "put $path: write did not verify (GET returned $HTTP)"
|
||||
fi
|
||||
echo "ok: PUT $path" ;;
|
||||
|
||||
post)
|
||||
[ $# -ge 1 ] || usage
|
||||
path="$1"; _capture_body "${2:-}"
|
||||
_curl POST "$(_vault_url "$path")" -H 'Content-Type: text/markdown' --data-binary @"$BODY"
|
||||
_check "post $path"; echo "ok: POST $path" ;;
|
||||
|
||||
append)
|
||||
# idempotent: GET the file, skip the POST if the exact line is already present.
|
||||
[ $# -ge 2 ] || usage
|
||||
path="$1"; line="$2"
|
||||
_curl GET "$(_vault_url "$path")"
|
||||
if [ "$HTTP" = "200" ] && grep -qF -- "$line" "$RESP"; then
|
||||
echo "skip: line already present in $path"; exit 0
|
||||
fi
|
||||
[ "$HTTP" = "200" ] || [ "$HTTP" = "404" ] || _check "append(read) $path"
|
||||
BODY="$(mktemp)"; printf '%s\n' "$line" > "$BODY"
|
||||
_curl POST "$(_vault_url "$path")" -H 'Content-Type: text/markdown' --data-binary @"$BODY"
|
||||
_check "append $path"; echo "ok: APPEND $path" ;;
|
||||
|
||||
patch)
|
||||
[ $# -ge 4 ] || usage
|
||||
path="$1"; op="$2"; ttype="$3"; target="$4"; _capture_body "${5:-}"
|
||||
case "$op" in append|prepend|replace) ;; *) die "patch: op must be append|prepend|replace";; esac
|
||||
case "$ttype" in heading|frontmatter|block) ;; *) die "patch: target-type must be heading|frontmatter|block";; esac
|
||||
_curl PATCH "$(_vault_url "$path")" \
|
||||
-H "Operation: $op" -H "Target-Type: $ttype" -H "Target: $target" \
|
||||
-H 'Content-Type: text/markdown' --data-binary @"$BODY"
|
||||
_check "patch $path ($target)"
|
||||
echo "ok: PATCH $op $ttype '$target' -> $path" ;;
|
||||
|
||||
fm)
|
||||
[ $# -ge 3 ] || usage
|
||||
path="$1"; field="$2"; value="$3"
|
||||
BODY="$(mktemp)"; printf '%s' "$value" > "$BODY"
|
||||
_curl PATCH "$(_vault_url "$path")" \
|
||||
-H 'Operation: replace' -H 'Target-Type: frontmatter' -H "Target: $field" \
|
||||
-H 'Content-Type: application/json' --data-binary @"$BODY"
|
||||
_check "fm $path.$field"; echo "ok: frontmatter $field -> $path" ;;
|
||||
|
||||
bump)
|
||||
[ $# -ge 1 ] || usage
|
||||
path="$1"; d="${2:-$(_today)}"
|
||||
exec "$0" fm "$path" updated "\"$d\"" ;;
|
||||
|
||||
delete)
|
||||
[ $# -ge 1 ] || usage
|
||||
_curl DELETE "$(_vault_url "$1")"; _check "delete $1"; echo "ok: DELETE $1" ;;
|
||||
|
||||
lock)
|
||||
# advisory lock: _agent/locks/vault.lock holds "<owner> @ <iso>". Honored cooperatively.
|
||||
[ $# -ge 1 ] || usage
|
||||
owner="$1"; lockpath="_agent/locks/vault.lock"
|
||||
_curl GET "$(_vault_url "$lockpath")"
|
||||
if [ "$HTTP" = "200" ] && [ -s "$RESP" ]; then
|
||||
cur="$(cat "$RESP")"; held_owner="${cur%% @ *}"; held_iso="${cur##* @ }"; held_iso="${held_iso%$'\n'}"
|
||||
held_epoch="$(date -u -j -f '%Y-%m-%dT%H:%M:%SZ' "$held_iso" +%s 2>/dev/null \
|
||||
|| date -u -d "$held_iso" +%s 2>/dev/null || echo 0)"
|
||||
now_epoch="$(date -u +%s)"
|
||||
if [ "$held_owner" != "$owner" ] && [ $((now_epoch - held_epoch)) -lt "$ECHO_LOCK_TTL" ]; then
|
||||
echo "lock held by '$held_owner' since $held_iso (fresh)" >&2; exit 75
|
||||
fi
|
||||
fi
|
||||
BODY="$(mktemp)"; printf '%s @ %s\n' "$owner" "$(_now_iso)" > "$BODY"
|
||||
_curl PUT "$(_vault_url "$lockpath")" -H 'Content-Type: text/markdown' --data-binary @"$BODY"
|
||||
_check "lock"; echo "ok: locked by $owner" ;;
|
||||
|
||||
unlock)
|
||||
[ $# -ge 1 ] || usage
|
||||
owner="$1"; lockpath="_agent/locks/vault.lock"
|
||||
_curl GET "$(_vault_url "$lockpath")"
|
||||
if [ "$HTTP" = "200" ]; then
|
||||
cur="$(cat "$RESP")"; held_owner="${cur%% @ *}"
|
||||
[ "$held_owner" = "$owner" ] || { echo "lock owned by '$held_owner', not '$owner' — not releasing" >&2; exit 75; }
|
||||
fi
|
||||
_curl DELETE "$(_vault_url "$lockpath")"
|
||||
[ "$HTTP" = "404" ] || _check "unlock"
|
||||
echo "ok: unlocked" ;;
|
||||
|
||||
""|-h|--help|help) usage ;;
|
||||
*) die "unknown command '$cmd' (try: get map ls search put post append patch fm bump delete lock unlock)" ;;
|
||||
esac
|
||||
@@ -0,0 +1,97 @@
|
||||
#!/usr/bin/env bash
|
||||
# migrate.sh — bring an existing ECHO vault up to the plugin's current schema.
|
||||
#
|
||||
# Reads the marker's schema_version and applies each intervening migration in order.
|
||||
# Migrations are idempotent and additive; every destructive step (DELETE) is gated
|
||||
# behind --apply AND prints what it will do first. Default mode is a DRY-RUN plan.
|
||||
#
|
||||
# Usage:
|
||||
# migrate.sh # print the migration plan (no changes)
|
||||
# migrate.sh --apply # perform the migration (moves/deletes included)
|
||||
#
|
||||
# Env: ECHO_BASE, ECHO_KEY (via echo.sh).
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ECHO="$SCRIPT_DIR/echo.sh"
|
||||
CURRENT_SCHEMA=2
|
||||
APPLY=0
|
||||
[ "${1:-}" = "--apply" ] && APPLY=1
|
||||
|
||||
[ -x "$ECHO" ] || chmod +x "$ECHO" 2>/dev/null || true
|
||||
say() { echo "migrate: $*"; }
|
||||
do_or_show() { # do_or_show "<human description>" cmd args...
|
||||
local desc="$1"; shift
|
||||
if [ "$APPLY" = "1" ]; then say "APPLY $desc"; "$@"; else say "PLAN $desc"; fi
|
||||
}
|
||||
|
||||
# ---- Read current schema -----------------------------------------------------
|
||||
if ! marker="$("$ECHO" get _agent/echo-vault.md 2>/dev/null)"; then
|
||||
say "marker missing — vault not bootstrapped. Run bootstrap.sh, not migrate.sh."
|
||||
exit 3
|
||||
fi
|
||||
FROM="$(printf '%s' "$marker" | sed -n 's/^schema_version:[[:space:]]*//p' | head -1)"
|
||||
FROM="${FROM:-0}"
|
||||
say "vault schema_version=$FROM, plugin schema=$CURRENT_SCHEMA $([ "$APPLY" = 1 ] && echo '(APPLY)' || echo '(dry-run)')"
|
||||
|
||||
if [ "$FROM" -ge "$CURRENT_SCHEMA" ] 2>/dev/null; then
|
||||
say "up to date — nothing to do."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
ls_files() { "$ECHO" ls "$1" 2>/dev/null | python3 -c 'import sys,json;print("\n".join(json.load(sys.stdin).get("files",[])))' 2>/dev/null || true; }
|
||||
move() { # move SRC DST preserving content (PUT dst <- get src, then delete src)
|
||||
local src="$1" dst="$2"
|
||||
"$ECHO" get "$src" 2>/dev/null | "$ECHO" put "$dst" - >/dev/null
|
||||
"$ECHO" delete "$src" >/dev/null
|
||||
}
|
||||
|
||||
# ---- 0 -> 1 : control docs moved into the plugin -----------------------------
|
||||
mig_0_1() {
|
||||
say "[0->1] retire in-vault control docs (CLAUDE/BOOTSTRAP/STRUCTURE/index.md)"
|
||||
for f in CLAUDE.md BOOTSTRAP.md STRUCTURE.md index.md; do
|
||||
if ECHO_VERIFY=0 "$ECHO" get "$f" >/dev/null 2>&1; then
|
||||
do_or_show "delete vault/$f (back it up outside the vault first)" "$ECHO" delete "$f"
|
||||
fi
|
||||
done
|
||||
say "[0->1] reminder: scrub dangling [[CLAUDE]]/[[BOOTSTRAP]]/[[STRUCTURE]]/[[index]] links from ## Related sections (manual/agent step)."
|
||||
}
|
||||
|
||||
# ---- 1 -> 2 : reviews/ folded into journal/ + _agent/health/ -----------------
|
||||
mig_1_2() {
|
||||
say "[1->2] fold reviews/ into journal/ and _agent/health/"
|
||||
for f in $(ls_files reviews/weekly); do
|
||||
[ "${f%.md}" != "$f" ] || continue
|
||||
dst="journal/weekly/$(printf '%s' "$f" | sed 's/-review\.md$/.md/')"
|
||||
do_or_show "move reviews/weekly/$f -> $dst" move "reviews/weekly/$f" "$dst"
|
||||
done
|
||||
for f in $(ls_files reviews/monthly); do
|
||||
[ "${f%.md}" != "$f" ] || continue
|
||||
case "$f" in
|
||||
*vault-health.md) dst="_agent/health/$f" ;;
|
||||
*) dst="journal/monthly/$f" ;;
|
||||
esac
|
||||
do_or_show "move reviews/monthly/$f -> $dst" move "reviews/monthly/$f" "$dst"
|
||||
done
|
||||
for period in quarterly annual; do
|
||||
for f in $(ls_files "reviews/$period"); do
|
||||
[ "${f%.md}" != "$f" ] || continue
|
||||
do_or_show "move reviews/$period/$f -> journal/$period/$f" move "reviews/$period/$f" "journal/$period/$f"
|
||||
done
|
||||
done
|
||||
say "[1->2] reminder: update inbound [[reviews/...]] wikilinks in ## Related sections (manual/agent step)."
|
||||
}
|
||||
|
||||
[ "$FROM" -lt 1 ] && mig_0_1
|
||||
[ "$FROM" -lt 2 ] && mig_1_2
|
||||
|
||||
# ---- Stamp the marker --------------------------------------------------------
|
||||
do_or_show "set _agent/echo-vault.md schema_version -> $CURRENT_SCHEMA" \
|
||||
"$ECHO" fm _agent/echo-vault.md schema_version "$CURRENT_SCHEMA"
|
||||
|
||||
if [ "$APPLY" = "1" ]; then
|
||||
say "migration complete -> schema $CURRENT_SCHEMA. Run vault-lint.sh to confirm invariants."
|
||||
else
|
||||
say "dry-run only. Re-run with --apply to perform the moves/deletes above."
|
||||
fi
|
||||
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"$comment": "CANONICAL machine-readable routing manifest for the ECHO vault. This is the single source of truth for 'what paths may be written to'. The human-readable tables in SKILL.md, references/routing-map.md, and references/api-reference.md are DERIVED views of this file — when they disagree, this file wins. vault-lint.sh consumes it to enforce the core rule: if a path matches no route here (and is not a retired path), nothing should be written to it. Patterns are Python regexes matched against vault-root-relative paths (no leading slash, no /vault/ prefix).",
|
||||
"schema_version": 2,
|
||||
"routes": [
|
||||
{ "id": "inbox-captures", "pattern": "^inbox/captures/inbox\\.md$", "method": "POST", "trigger": "Destination unknown at capture time", "distinct_because": "Only path whose contract is deferred routing" },
|
||||
{ "id": "inbox-imports", "pattern": "^inbox/imports/[^/]+\\.md$", "method": "PUT", "trigger": "Raw external material dropped wholesale", "distinct_because": "Bulk un-triaged material vs single-line captures" },
|
||||
{ "id": "inbox-processing-log", "pattern": "^inbox/processing-log/\\d{4}-\\d{2}-\\d{2}\\.md$", "method": "POST", "trigger": "An inbox item is routed to its real home", "distinct_because": "Audit trail of moves, not memory itself" },
|
||||
|
||||
{ "id": "journal-daily", "pattern": "^journal/daily/\\d{4}-\\d{2}-\\d{2}\\.md$", "method": "PATCH", "trigger": "First agent activity on a given day", "distinct_because": "Finest grain; PATCHed repeatedly within its period" },
|
||||
{ "id": "journal-weekly", "pattern": "^journal/weekly/\\d{4}-W\\d{2}\\.md$", "method": "PUT", "trigger": "First substantive session of a new ISO week (opt-in)", "distinct_because": "ISO-week grain" },
|
||||
{ "id": "journal-monthly", "pattern": "^journal/monthly/\\d{4}-\\d{2}\\.md$", "method": "PUT", "trigger": "First substantive session of a new month", "distinct_because": "Month grain" },
|
||||
{ "id": "journal-quarterly", "pattern": "^journal/quarterly/\\d{4}-Q[1-4]\\.md$", "method": "PUT", "trigger": "Manual / on request only", "distinct_because": "Strategic grain; never auto-fires" },
|
||||
{ "id": "journal-annual", "pattern": "^journal/annual/\\d{4}\\.md$", "method": "PUT", "trigger": "Manual / on request only", "distinct_because": "Coarsest grain; never auto-fires" },
|
||||
{ "id": "journal-templates", "pattern": "^journal/templates/.+\\.md$", "method": "PUT", "trigger": "Bootstrap seed only", "distinct_because": "Holds templates, not journal content" },
|
||||
|
||||
{ "id": "projects-active", "pattern": "^projects/active/[^/]+\\.md$", "method": "PUT", "trigger": "Work in motion now", "distinct_because": "Default state for anything being worked", "status": "active" },
|
||||
{ "id": "projects-incubating", "pattern": "^projects/incubating/[^/]+\\.md$", "method": "PUT", "trigger": "Idea captured, work not started", "distinct_because": "Pre-work", "status": "incubating" },
|
||||
{ "id": "projects-on-hold", "pattern": "^projects/on-hold/[^/]+\\.md$", "method": "PUT", "trigger": "Paused but still tracked", "distinct_because": "Resumable; not terminal", "status": "on-hold" },
|
||||
{ "id": "projects-archived", "pattern": "^projects/archived/[^/]+\\.md$", "method": "PUT", "trigger": "Done, abandoned, or rolled up", "distinct_because": "Terminal; kept for history", "status": "archived" },
|
||||
{ "id": "projects-template", "pattern": "^projects/project-template\\.md$", "method": "PUT", "trigger": "Bootstrap seed only", "distinct_because": "Template, not a project" },
|
||||
|
||||
{ "id": "areas", "pattern": "^areas/(business|personal|learning|systems)/[^/]+\\.md$", "method": "PUT", "trigger": "Ongoing domain with no finish line", "distinct_because": "No end state — disqualifies it from projects/" },
|
||||
|
||||
{ "id": "resources-people", "pattern": "^resources/people/[^/]+\\.md$", "method": "PUT", "trigger": "A fact about a specific person", "distinct_because": "Keyed to a person" },
|
||||
{ "id": "resources-companies", "pattern": "^resources/companies/[^/]+\\.md$", "method": "PUT", "trigger": "A fact about an organization", "distinct_because": "Keyed to an organization, not an individual" },
|
||||
{ "id": "resources-concepts", "pattern": "^resources/concepts/[^/]+\\.md$", "method": "PUT", "trigger": "A reusable concept/idea", "distinct_because": "An idea vs an external source" },
|
||||
{ "id": "resources-references", "pattern": "^resources/references/[^/]+\\.md$", "method": "PUT", "trigger": "An external source/link worth keeping", "distinct_because": "Points outward" },
|
||||
{ "id": "resources-meetings", "pattern": "^resources/meetings/\\d{4}-\\d{2}-\\d{2}-[^/]+\\.md$", "method": "PUT", "trigger": "Notes tied to a specific meeting", "distinct_because": "Event-anchored to a meeting" },
|
||||
|
||||
{ "id": "decisions-by-date", "pattern": "^decisions/by-date/\\d{4}-\\d{2}-\\d{2}-[^/]+\\.md$", "method": "PUT", "trigger": "A non-obvious decision worth recording", "distinct_because": "Chronological system of record for decisions" },
|
||||
{ "id": "decisions-template", "pattern": "^decisions/decision-template\\.md$", "method": "PUT", "trigger": "Bootstrap seed only", "distinct_because": "Template, not a decision" },
|
||||
|
||||
{ "id": "agent-marker", "pattern": "^_agent/echo-vault\\.md$", "method": "PUT", "trigger": "Bootstrap / schema migration only", "distinct_because": "Plugin-owned probe; never hand-edited" },
|
||||
{ "id": "agent-context", "pattern": "^_agent/context/[^/]+\\.md$", "method": "PATCH", "trigger": "Active scope changes / task bundles", "distinct_because": "Single live scope pointer + bundles" },
|
||||
{ "id": "agent-semantic", "pattern": "^_agent/memory/semantic/[^/]+\\.md$", "method": "PUT", "trigger": "A durable fact/pattern (incl. operator-preferences.md)", "distinct_because": "Timeless fact" },
|
||||
{ "id": "agent-episodic", "pattern": "^_agent/memory/episodic/[^/]+\\.md$", "method": "PUT", "trigger": "A record of what happened, when", "distinct_because": "Anchored to an event in time" },
|
||||
{ "id": "agent-working", "pattern": "^_agent/memory/working/[^/]+\\.md$", "method": "PUT", "trigger": "Short-lived state for the current effort", "distinct_because": "Explicitly transient" },
|
||||
{ "id": "agent-sessions", "pattern": "^_agent/sessions/\\d{4}-\\d{2}-\\d{2}(-\\d{4})?-[^/]+\\.md$", "method": "PUT", "trigger": "A substantive session ends", "distinct_because": "Per-session record (new ones require HHMM)" },
|
||||
{ "id": "agent-health", "pattern": "^_agent/health/\\d{4}-\\d{2}-vault-health\\.md$", "method": "PUT", "trigger": "First substantive session of a month", "distinct_because": "Vault integrity, not work narrative" },
|
||||
{ "id": "agent-heartbeat", "pattern": "^_agent/heartbeat/[^/]+\\.md$", "method": "PUT", "trigger": "End of every session", "distinct_because": "O(1) orientation pointer; overwritten, never grows" },
|
||||
{ "id": "agent-templates", "pattern": "^_agent/templates/.+\\.md$", "method": "PUT", "trigger": "Bootstrap seed only", "distinct_because": "Holds templates, not memory" },
|
||||
{ "id": "agent-skills-active", "pattern": "^_agent/skills/active/[^/]+\\.md$", "method": "PUT", "trigger": "A skill/plugin catalogued as a capability", "distinct_because": "Catalogs a capability vs the build effort" },
|
||||
{ "id": "agent-skills-archived","pattern": "^_agent/skills/archived/[^/]+\\.md$", "method": "PUT", "trigger": "A catalogued skill is retired", "distinct_because": "Terminal state of the skill catalog" },
|
||||
{ "id": "agent-locks", "pattern": "^_agent/locks/[^/]+\\.lock$", "method": "PUT", "trigger": "Advisory multi-writer lock acquire/release", "distinct_because": "Concurrency coordination, not memory" },
|
||||
|
||||
{ "id": "leaf-readme", "pattern": "^(.+/)?README\\.md$", "method": "PUT", "trigger": "Bootstrap leaf signpost / vault root README", "distinct_because": "Human signpost, not read for routing" }
|
||||
],
|
||||
"retired": [
|
||||
{ "pattern": "^reviews/", "retired_in_schema": 2, "replacement": "journal/{weekly,monthly,quarterly,annual}/ and _agent/health/" },
|
||||
{ "pattern": "^decisions/by-project/", "retired_in_schema": 1, "replacement": "[[wikilink]] under the project's ## Key Decisions" },
|
||||
{ "pattern": "^archive/", "retired_in_schema": 0, "replacement": "projects/archived/ and _agent/skills/archived/" },
|
||||
{ "pattern": "^(CLAUDE|BOOTSTRAP|STRUCTURE|index)\\.md$", "retired_in_schema": 1, "replacement": "All control logic lives in the plugin references/, not the vault" }
|
||||
]
|
||||
}
|
||||
Regular → Executable
+177
-59
@@ -1,27 +1,37 @@
|
||||
#!/usr/bin/env bash
|
||||
# vault-lint.sh — mechanically assert ECHO vault invariants.
|
||||
#
|
||||
# Catches the recurring "invariant violation" bugs that prose rules can't enforce
|
||||
# on their own: folder<->status drift, duplicate slugs, wikilinks leaking into
|
||||
# frontmatter, duplicate "## Agent Log" headings, stale active projects, and
|
||||
# aging inbox captures. Invoked by the monthly Vault Health pass (see SKILL.md),
|
||||
# but safe to run any time — it is READ-ONLY and never modifies the vault.
|
||||
# Catches the recurring "invariant violation" bugs that prose rules can't enforce:
|
||||
# folder<->status drift, duplicate slugs, wikilinks leaking into frontmatter,
|
||||
# duplicate "## Agent Log" headings, stale active projects, aging inbox captures,
|
||||
# impossible dates, bad status values, missing frontmatter, broken source_notes, and
|
||||
# paths that no route in routing.json permits. Invoked by the monthly Vault Health
|
||||
# pass (see SKILL.md), but safe to run any time — it is READ-ONLY.
|
||||
#
|
||||
# Exit status: 0 = clean, 1 = violations found, 2 = vault unreachable.
|
||||
# Exit status: 0 = clean, 1 = violations found, 2 = vault unreachable,
|
||||
# 3 = vault not bootstrapped (marker missing).
|
||||
#
|
||||
# Config is hardcoded to match the rest of the plugin; override via env if needed:
|
||||
# Config (env overrides):
|
||||
# ECHO_BASE (default https://echoapi.alwisp.com)
|
||||
# ECHO_KEY (default the plugin's bearer token)
|
||||
# ECHO_TODAY (default the machine date) — pass the conversation's currentDate so
|
||||
# stale/aging math uses the SAME clock the agent writes with (YYYY-MM-DD)
|
||||
# STALE_DAYS (default 30) INBOX_DAYS (default 14)
|
||||
#
|
||||
# routing.json (canonical route manifest) is read from this script's own directory.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
ECHO_BASE="${ECHO_BASE:-https://echoapi.alwisp.com}"
|
||||
ECHO_KEY="${ECHO_KEY:-241265fbe6830934a9a4ad3e69335f64a42153b663aa5b0017cb1ea1217b2bab}"
|
||||
STALE_DAYS="${STALE_DAYS:-30}"
|
||||
INBOX_DAYS="${INBOX_DAYS:-14}"
|
||||
ECHO_TODAY="${ECHO_TODAY:-$(date +%Y-%m-%d)}"
|
||||
|
||||
ECHO_BASE="$ECHO_BASE" ECHO_KEY="$ECHO_KEY" STALE_DAYS="$STALE_DAYS" INBOX_DAYS="$INBOX_DAYS" \
|
||||
ECHO_TODAY="$ECHO_TODAY" ROUTING_JSON="$SCRIPT_DIR/routing.json" \
|
||||
python3 - <<'PY'
|
||||
import os, sys, json, re, datetime, urllib.request, urllib.error
|
||||
|
||||
@@ -29,9 +39,21 @@ BASE = os.environ["ECHO_BASE"].rstrip("/")
|
||||
KEY = os.environ["ECHO_KEY"]
|
||||
STALE_DAYS = int(os.environ["STALE_DAYS"])
|
||||
INBOX_DAYS = int(os.environ["INBOX_DAYS"])
|
||||
TODAY = datetime.date.today()
|
||||
TODAY = datetime.date.fromisoformat(os.environ["ECHO_TODAY"])
|
||||
ROUTING_JSON = os.environ["ROUTING_JSON"]
|
||||
LIFECYCLES = ["active", "incubating", "on-hold", "archived"]
|
||||
SKIP = {"README.md", "project-template.md", "decision-template.md"}
|
||||
REQUIRED_FM = ("type", "created")
|
||||
# Project status vocabulary IS enforced (status must equal the lifecycle folder) by the
|
||||
# folder/status check below. Other note kinds (decisions/concepts) carry free-form status
|
||||
# vocab (accepted, shipped, reference, ...), so there is no global status allow-list.
|
||||
|
||||
# optional real YAML parser; fall back to a tolerant line parser
|
||||
try:
|
||||
import yaml # type: ignore
|
||||
HAVE_YAML = True
|
||||
except Exception:
|
||||
HAVE_YAML = False
|
||||
|
||||
violations = []
|
||||
def flag(check, msg): violations.append((check, msg))
|
||||
@@ -48,32 +70,72 @@ def get(path):
|
||||
return None
|
||||
raise
|
||||
|
||||
def listdir(path):
|
||||
body = get(path if path.endswith("/") else path + "/")
|
||||
if body is None:
|
||||
return []
|
||||
def list_dir(path):
|
||||
"""Return (files, folders) for a vault directory. Directories may arrive either in a
|
||||
'folders' key OR as 'files' entries ending in '/'; handle both. Root is '' -> /vault/.
|
||||
Tolerates non-404 errors (e.g. a 400 on an odd path) by returning empty."""
|
||||
p = "" if path in ("", "/") else (path if path.endswith("/") else path + "/")
|
||||
try:
|
||||
return json.loads(body).get("files", [])
|
||||
body = get(p)
|
||||
except urllib.error.HTTPError:
|
||||
return [], []
|
||||
if body is None:
|
||||
return [], []
|
||||
try:
|
||||
j = json.loads(body)
|
||||
except json.JSONDecodeError:
|
||||
return []
|
||||
return [], []
|
||||
entries = list(j.get("files", [])) + list(j.get("folders", []))
|
||||
files = [e for e in entries if not e.endswith("/")]
|
||||
folders = [e[:-1] for e in entries if e.endswith("/")]
|
||||
return files, folders
|
||||
|
||||
def frontmatter(text):
|
||||
"""Return (raw_frontmatter_str, dict_of_scalar_fields). Empty if no block."""
|
||||
if not text or not text.startswith("---"):
|
||||
def walk(prefix=""):
|
||||
"""Yield every file path under prefix (recursive). prefix is '' or ends with '/'."""
|
||||
files, folders = list_dir(prefix)
|
||||
for f in files:
|
||||
yield prefix + f
|
||||
for d in folders:
|
||||
yield from walk(f"{prefix}{d}/")
|
||||
|
||||
def split_frontmatter(text):
|
||||
"""Return (raw_yaml_str, body) splitting on anchored ^---$ delimiters. ('', text) if none."""
|
||||
if not text:
|
||||
return "", ""
|
||||
lines = text.splitlines()
|
||||
if not lines or lines[0].strip() != "---":
|
||||
return "", text
|
||||
for i in range(1, len(lines)):
|
||||
if lines[i].strip() == "---":
|
||||
return "\n".join(lines[1:i]), "\n".join(lines[i+1:])
|
||||
return "", text # unterminated block -> treat as no frontmatter
|
||||
|
||||
def parse_fm(text):
|
||||
"""Return (raw_yaml_str, dict). Uses PyYAML when available, else a tolerant parser."""
|
||||
raw, _ = split_frontmatter(text)
|
||||
if not raw:
|
||||
return "", {}
|
||||
end = text.find("\n---", 3)
|
||||
if end == -1:
|
||||
return "", {}
|
||||
raw = text[3:end]
|
||||
if HAVE_YAML:
|
||||
try:
|
||||
d = yaml.safe_load(raw)
|
||||
return raw, (d if isinstance(d, dict) else {})
|
||||
except Exception:
|
||||
pass
|
||||
# fallback: scalar + simple inline-list lines (keys may contain digits, _, -)
|
||||
fields = {}
|
||||
for line in raw.splitlines():
|
||||
m = re.match(r"^([A-Za-z_]+):\s*(.*)$", line)
|
||||
m = re.match(r"^([A-Za-z_][\w-]*):\s*(.*)$", line)
|
||||
if m:
|
||||
fields[m.group(1)] = m.group(2).strip()
|
||||
v = m.group(2).strip()
|
||||
if v.startswith("[") and v.endswith("]"):
|
||||
v = [x.strip().strip('"').strip("'") for x in v[1:-1].split(",") if x.strip()]
|
||||
else:
|
||||
v = v.strip('"').strip("'")
|
||||
fields[m.group(1)] = v
|
||||
return raw, fields
|
||||
|
||||
def parse_date(s):
|
||||
m = re.match(r"(\d{4}-\d{2}-\d{2})", s or "")
|
||||
m = re.match(r"(\d{4}-\d{2}-\d{2})", str(s or ""))
|
||||
if not m:
|
||||
return None
|
||||
try:
|
||||
@@ -81,66 +143,115 @@ def parse_date(s):
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
# Reachability probe
|
||||
def as_list(v):
|
||||
if v is None or v == "":
|
||||
return []
|
||||
return v if isinstance(v, list) else [v]
|
||||
|
||||
# ---- Reachability + bootstrap probe (M2: do NOT silently report clean) -------
|
||||
try:
|
||||
if get("_agent/echo-vault.md") is None:
|
||||
print("vault-lint: marker missing — vault may not be bootstrapped.", file=sys.stderr)
|
||||
print("vault-lint: marker missing — vault not bootstrapped (run bootstrap.sh).", file=sys.stderr)
|
||||
sys.exit(3)
|
||||
except Exception as e:
|
||||
print(f"vault-lint: vault unreachable ({e}).", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
# ---- Projects: folder<->status, stale active, wikilinks-in-frontmatter, dup slugs
|
||||
# ---- Load canonical routing manifest (S3) ------------------------------------
|
||||
ROUTES, RETIRED = [], []
|
||||
try:
|
||||
with open(ROUTING_JSON) as fh:
|
||||
rj = json.load(fh)
|
||||
ROUTES = [(r["id"], re.compile(r["pattern"])) for r in rj.get("routes", [])]
|
||||
RETIRED = [(re.compile(r["pattern"]), r.get("replacement", "")) for r in rj.get("retired", [])]
|
||||
except Exception as e:
|
||||
flag("routing-manifest", f"could not load routing.json ({e}) — path checks skipped")
|
||||
|
||||
# ---- Single full walk feeds every path-level check ---------------------------
|
||||
all_files = list(walk())
|
||||
|
||||
def route_for(path):
|
||||
for rid, rx in ROUTES:
|
||||
if rx.match(path):
|
||||
return rid
|
||||
return None
|
||||
|
||||
# Path membership + retired-path detection (S3)
|
||||
for path in all_files:
|
||||
if ROUTES and route_for(path) is None:
|
||||
hit = next((repl for rx, repl in RETIRED if rx.match(path)), None)
|
||||
if hit is not None:
|
||||
flag("retired-path", f"{path}: retired location — should be {hit}")
|
||||
else:
|
||||
flag("unknown-path", f"{path}: matches no route in routing.json")
|
||||
|
||||
# ---- Per-note frontmatter checks (M5) ----------------------------------------
|
||||
TEMPLATE_RE = re.compile(r"(^|/)(templates/|.*-template\.md$)")
|
||||
for path in all_files:
|
||||
base = path.rsplit("/", 1)[-1]
|
||||
if base in SKIP or TEMPLATE_RE.search(path) or not path.endswith(".md"):
|
||||
continue
|
||||
text = get(path)
|
||||
if text is None:
|
||||
continue
|
||||
raw, fm = parse_fm(text)
|
||||
|
||||
# wikilinks anywhere in frontmatter (widened sweep — all folders)
|
||||
if "[[" in raw:
|
||||
flag("frontmatter-wikilink", f"{path}: '[[...]]' inside frontmatter")
|
||||
|
||||
# missing required frontmatter
|
||||
missing = [k for k in REQUIRED_FM if not str(fm.get(k, "")).strip()]
|
||||
if fm and missing:
|
||||
flag("missing-frontmatter", f"{path}: missing {', '.join(missing)}")
|
||||
|
||||
# impossible dates: updated < created
|
||||
c, u = parse_date(fm.get("created")), parse_date(fm.get("updated"))
|
||||
if c and u and u < c:
|
||||
flag("date-order", f"{path}: updated {u} is before created {c}")
|
||||
if u and u > TODAY:
|
||||
flag("future-date", f"{path}: updated {u} is in the future (today {TODAY})")
|
||||
|
||||
# source_notes hygiene: plain relative paths, never wikilinks, no self-reference
|
||||
for sn in as_list(fm.get("source_notes")):
|
||||
s = str(sn)
|
||||
if "[[" in s:
|
||||
flag("source-notes-wikilink", f"{path}: source_notes contains a wikilink '{s}'")
|
||||
|
||||
# ---- Projects: folder<->status, stale active, duplicate slugs ----------------
|
||||
slug_homes = {}
|
||||
for lc in LIFECYCLES:
|
||||
for fn in listdir(f"projects/{lc}"):
|
||||
if fn.endswith("/") or fn in SKIP:
|
||||
files, _ = list_dir(f"projects/{lc}")
|
||||
for fn in files:
|
||||
if fn.endswith("/") or fn in SKIP or not fn.endswith(".md"):
|
||||
continue
|
||||
slug = fn[:-3] if fn.endswith(".md") else fn
|
||||
slug = fn[:-3]
|
||||
slug_homes.setdefault(slug, []).append(lc)
|
||||
text = get(f"projects/{lc}/{fn}")
|
||||
if text is None:
|
||||
continue
|
||||
raw, fm = frontmatter(text)
|
||||
status = fm.get("status", "").strip().strip('"').strip("'")
|
||||
_, fm = parse_fm(text)
|
||||
status = str(fm.get("status", "")).strip().strip('"').strip("'")
|
||||
if status and status != lc:
|
||||
flag("folder/status", f"projects/{lc}/{fn}: status='{status}' but folder='{lc}'")
|
||||
if "[[" in raw:
|
||||
flag("frontmatter-wikilink", f"projects/{lc}/{fn}: '[[...]]' inside frontmatter")
|
||||
if lc == "active":
|
||||
d = parse_date(fm.get("updated", ""))
|
||||
d = parse_date(fm.get("updated"))
|
||||
if d and (TODAY - d).days > STALE_DAYS:
|
||||
flag("stale-active", f"projects/active/{fn}: updated {d} ({(TODAY-d).days}d ago) — consider on-hold/")
|
||||
for slug, homes in slug_homes.items():
|
||||
if len(homes) > 1:
|
||||
flag("duplicate-slug", f"'{slug}' exists in {', '.join(homes)}")
|
||||
|
||||
# ---- Wikilinks-in-frontmatter for other high-churn notes
|
||||
extra = ["_agent/context/current-context.md",
|
||||
"_agent/memory/semantic/operator-preferences.md"]
|
||||
for fn in listdir("resources/people"):
|
||||
if fn.endswith(".md") and fn not in SKIP:
|
||||
extra.append(f"resources/people/{fn}")
|
||||
for fn in listdir("_agent/memory/semantic"):
|
||||
if fn.endswith(".md") and fn not in SKIP:
|
||||
extra.append(f"_agent/memory/semantic/{fn}")
|
||||
for path in extra:
|
||||
text = get(path)
|
||||
if text is None:
|
||||
# ---- Daily notes: duplicate "## Agent Log" headings --------------------------
|
||||
for path in all_files:
|
||||
if not re.match(r"^journal/daily/.*\.md$", path):
|
||||
continue
|
||||
raw, _ = frontmatter(text)
|
||||
if "[[" in raw:
|
||||
flag("frontmatter-wikilink", f"{path}: '[[...]]' inside frontmatter")
|
||||
|
||||
# ---- Daily notes: duplicate "## Agent Log" headings
|
||||
for fn in listdir("journal/daily"):
|
||||
if not fn.endswith(".md") or fn in SKIP:
|
||||
continue
|
||||
text = get(f"journal/daily/{fn}") or ""
|
||||
text = get(path) or ""
|
||||
n = len(re.findall(r"(?m)^## Agent Log\s*$", text))
|
||||
if n > 1:
|
||||
flag("duplicate-agent-log", f"journal/daily/{fn}: {n} '## Agent Log' headings")
|
||||
flag("duplicate-agent-log", f"{path}: {n} '## Agent Log' headings")
|
||||
|
||||
# ---- Inbox: captures aging past INBOX_DAYS
|
||||
# ---- Inbox: captures aging past INBOX_DAYS -----------------------------------
|
||||
inbox = get("inbox/captures/inbox.md") or ""
|
||||
for line in inbox.splitlines():
|
||||
m = re.match(r"^\s*-\s*(\d{4}-\d{2}-\d{2})\b", line)
|
||||
@@ -149,7 +260,7 @@ for line in inbox.splitlines():
|
||||
if d and (TODAY - d).days > INBOX_DAYS:
|
||||
flag("aging-inbox", f"inbox capture {d} ({(TODAY-d).days}d): {line.strip()[:80]}")
|
||||
|
||||
# ---- Report
|
||||
# ---- Report ------------------------------------------------------------------
|
||||
if not violations:
|
||||
print("vault-lint: clean — all invariants hold.")
|
||||
sys.exit(0)
|
||||
@@ -165,6 +276,13 @@ labels = {
|
||||
"duplicate-agent-log": "Duplicate '## Agent Log' heading",
|
||||
"stale-active": f"Stale active project (updated > {STALE_DAYS}d)",
|
||||
"aging-inbox": f"Inbox capture aging (> {INBOX_DAYS}d)",
|
||||
"unknown-path": "Path matches no route in routing.json",
|
||||
"retired-path": "Write to a retired/dead path",
|
||||
"missing-frontmatter": "Missing required frontmatter field",
|
||||
"date-order": "updated earlier than created",
|
||||
"future-date": "updated date is in the future",
|
||||
"source-notes-wikilink": "Wikilink in source_notes (must be plain paths)",
|
||||
"routing-manifest": "routing.json problem",
|
||||
}
|
||||
for check, msgs in by.items():
|
||||
print(f"## {labels.get(check, check)}")
|
||||
|
||||
Reference in New Issue
Block a user