diff --git a/README.md b/README.md new file mode 100644 index 0000000..c2d0e11 --- /dev/null +++ b/README.md @@ -0,0 +1,299 @@ +# echo-memory — v0.5 + +Persistent memory for Claude / CoWork sessions via the **ECHO** Obsidian vault, driven over the [Obsidian Local REST API](https://github.com/coddingtonbear/obsidian-local-rest-api). No MCP server — the skill makes direct REST calls. + +Built for **Jason Stedwell** (Director of Technical Services / Systems Engineer, MPM / ALABAMA wISP), who is both the **operator** and the **architect** of this vault. This is a personal plugin, not for distribution. + +This repository (`jason/echo-v.05`) holds the **v0.5 source** of the plugin: the tracked source tree at `echo-memory.plugin.src/` and the built `echo-memory.plugin` package artifact rebuilt from it on each version bump. + +--- + +## Core design principle — the plugin is the single source of truth + +Everything that defines how ECHO behaves — bootstrap/repair logic, the operating contract, the taxonomy, frontmatter conventions, the routing map, and the canonical note templates — ships **inside the plugin** under `skills/echo-memory/`. + +The vault itself holds **data only**. There are no `CLAUDE.md` / `BOOTSTRAP.md` / `STRUCTURE.md` / `index.md` control docs in it. The single in-vault control artifact is a one-line marker, `_agent/echo-vault.md`, that records the schema version. + +Three consequences follow: + +- **Self-bootstrapping** — point the REST API at an empty Obsidian vault and the plugin stands up the full folder tree, templates, anchors, and marker from its bundled `scaffold/`. +- **Easy to update** — change behavior by updating the plugin, not by editing files scattered through the vault. +- **Portable** — the same plugin brings any empty vault online identically; nothing essential depends on files pre-existing in the vault. + +--- + +## Configuration + +The plugin is hardcoded for a single personal endpoint: + +- **Server:** `https://echoapi.alwisp.com` (reverse proxy → backend Obsidian Local REST API). This is the **only** valid endpoint — never use LAN addresses (`10.x`, `192.168.x`, `:27124`); those belong to retired memory systems. +- **Auth:** a bearer token stored in the skill / `references`. The key lives only in the plugin — **never inside a vault note** (per the vault's own safety rules). +- **TLS:** the endpoint presents a valid certificate, so `-k` is not needed. + +Vault paths are addressed **at the root** (e.g. `GET /vault/_agent/context/current-context.md`). + +--- + +## Requirements + +- Obsidian running on the backend with the Local REST API plugin enabled. +- HTTPS reachability to `https://echoapi.alwisp.com` from the Claude / CoWork session environment. + +--- + +## Repository layout + +``` +echo-v.05/ +├── echo-memory.plugin ← built, installable plugin (zip artifact, rebuilt on version bump) +└── echo-memory.plugin.src/ ← tracked source tree + ├── .claude-plugin/plugin.json ← manifest (name, version, description) + ├── README.md ← plugin-level README + └── skills/echo-memory/ + ├── SKILL.md ← operating procedure (authoritative) + ├── references/ + │ ├── operating-contract.md ← durable principles + safety rules + │ ├── bootstrap.md ← bootstrap / repair / migrate manifest + │ ├── vault-layout.md ← canonical layout + frontmatter conventions + │ ├── api-reference.md ← REST endpoint patterns + routing map + │ └── session-log-template.md + └── scaffold/ ← verbatim files the bootstrap writes into the vault + ├── echo-vault.md ← bootstrap marker + ├── README.vault.md ← thin human signpost + ├── anchors/ ← operator-preferences, current-context, inbox seeds + └── templates/ ← 8 canonical note templates +``` + +**Division of responsibility:** `SKILL.md` owns day-to-day *procedure* (loading order, search-first, triage, scope switching, PATCH rules). `references/operating-contract.md` owns the durable, client-independent *principles and safety rules*. The other references are the canonical layout, API, and bootstrap specs. + +--- + +## Vault layout (data only, root-addressed) + +``` +/vault/ +├── README.md ← thin human signpost (NOT read for routing) +├── inbox/ +│ ├── captures/ ← quick captures (inbox.md), date-prefixed lines +│ ├── imports/ ← raw imported material +│ └── processing-log/ +├── journal/ +│ ├── daily/ ← YYYY-MM-DD.md (has an "## Agent Log" section) +│ └── templates/ +├── projects/ ← lifecycle: incubating → active → on-hold / archived +│ ├── active/ incubating/ on-hold/ archived/ +│ └── project-template.md +├── areas/ ← business / personal / learning / systems +├── resources/ +│ ├── concepts/ references/ meetings/ +│ └── people/ ← .md +├── decisions/ +│ ├── by-date/ ← YYYY-MM-DD-.md (ADR-style) — the canonical home +│ └── decision-template.md (by-project/ is retired, not created) +├── reviews/ ← weekly / monthly / quarterly / annual +└── _agent/ + ├── echo-vault.md ← bootstrap marker: schema_version + date (plugin-owned probe) + ├── context/ ← current-context.md and task bundles + ├── memory/ + │ ├── working/ ← transient, time-boxed + │ ├── episodic/ ← what happened, when + │ └── semantic/ ← durable facts / patterns (operator-preferences.md) + ├── sessions/ ← YYYY-MM-DD-HHMM-.md + ├── templates/ ← canonical note templates (seeded from the plugin's scaffold/) + ├── skills/ ← active / archived capability catalog + └── heartbeat/ ← single-line pointers (e.g. last-session.md) +``` + +### Memory model + +| Layer | Path | What it holds | +|-------|------|---------------| +| Working | `_agent/memory/working/` | Transient, time-boxed state | +| Episodic | `_agent/memory/episodic/` | What happened, when | +| Semantic | `_agent/memory/semantic/` | Durable facts, patterns, preferences (`operator-preferences.md`) | +| Context | `_agent/context/` | Task-focused reading lists + active scope | + +--- + +## Operating procedures (logic) + +### Cold-start loading + +Load memory at the start of any substantive conversation (anything beyond a single quick factual question). The cold-start reads are **issued in parallel** (one batch of 4–5 GETs): + +1. `_agent/echo-vault.md` — the bootstrap marker. `404` → vault not set up (run bootstrap); `200` → check `schema_version` and migrate if older than the plugin's. +2. `_agent/memory/semantic/operator-preferences.md` — Jason's profile. +3. `_agent/context/current-context.md` — active scope + Scope History. +4. `_agent/sessions/` listing — read the ~5 most recent by reverse lexical sort (filenames lex-sort chronologically). +5. `journal/daily/YYYY-MM-DD.md` — today's note (`404` is fine). + +If a specific project is in play, follow up with a `search/simple/` across **all** lifecycle subfolders, querying **both the slug and any human title** used in conversation, then read the match at `projects//.md`. Loading is not narrated to the operator. + +### Inbox triage + +`inbox/captures/inbox.md` is the catch-all. At session start, GET it; if it holds lines older than ~7 days that were never routed, surface a count once and offer to triage. Accepted items route to their proper home (preference → operator-preferences; project idea → `projects/incubating/`; durable fact → semantic; person fact → `resources/people/`), with the move recorded one-line in `inbox/processing-log/YYYY-MM-DD.md` as an audit trail. Originals aren't deleted unless explicitly asked. + +### Project lifecycle + +Projects move through four folders; the folder name and the `status:` frontmatter field **MUST agree** — they are two views of one state. A file in `projects/active/` with `status: on-hold` is broken state. + +| Folder | `status:` | Meaning | +|--------|-----------|---------| +| `incubating/` | `incubating` | Idea captured; not actively worked | +| `active/` | `active` | Current work (default for anything in motion) | +| `on-hold/` | `on-hold` | Paused but still tracked; resumable | +| `archived/` | `archived` | Done, abandoned, or rolled up — not deleted | + +Promotion = move the file **and** update `status:` in the same change. + +### Scope switching + +`_agent/context/current-context.md` tracks one active scope. When scope changes: (1) PATCH-prepend a dated bullet capturing the **prior** scope to `## Scope History`; (2) PATCH-replace `## Scope` with the new scope; (3) bump the frontmatter `updated:`. Scope History is trimmed to the last ~10 entries. + +### Daily note — Agent Log + +After substantive activity, append a one-line entry to today's daily note's `## Agent Log`. The procedure is resilient: GET the note → if `404`, PUT it from the template → if `200` but the heading is missing, POST the heading. **Heading detection greps the raw markdown for an anchored `^## Agent Log`** — not the document-map JSON, whose headings are `::`-delimited paths (a bug fixed in 0.4.1 that previously appended duplicate headings). + +### Session logging + +At the end of substantive conversations (those producing decisions, artifacts, or commitments), write a session log to `_agent/sessions/YYYY-MM-DD-HHMM-.md`. **The four-digit HHMM component is canonical, not optional** — it makes filenames lex-sort in true chronological order, which cold-start loading depends on. New writes are validated against `^\d{4}-\d{2}-\d{2}-\d{4}-[a-z0-9-]+\.md$`. + +### Reviews + +- **Weekly** — opt-in; offered on the first substantive session of a new ISO week, written only if accepted, to `reviews/weekly/YYYY-Www-review.md`. +- **Monthly** — automatic Vault Health pass (below) on the first session of a calendar month. +- **Quarterly / annual** — manual / on request only. + +### Vault Health (monthly) + +A cheap pass written to `reviews/monthly/YYYY-MM-vault-health.md`: flag stale active projects (`updated:` > 30 days), unprocessed inbox items (> 14 days), duplicate slugs across lifecycle folders (broken state), and broken-heading risk on frequently-PATCHed files. Findings are reported, not auto-fixed. + +--- + +## Write-safety rules + +- **Search first (mandatory for new notes).** Before creating any slug-addressed note, `POST /search/simple/?query=` across the **whole** vault (and search the human title too) — listing one folder misses duplicates elsewhere. On a match: promote/merge from a non-active subfolder, update-in-place in the same folder, or ask which is canonical if elsewhere. +- **Read before append (idempotency).** `POST` is not idempotent — a retry duplicates lines. Before any POST that adds an entry (inbox, Agent Log, an Observations/Log heading), GET the target and skip if the exact line is already present. (Does not apply to PUT or PATCH-replace.) +- **Bump `updated:` on substance.** PATCH the frontmatter `updated:` to today after a meaningful content change (status update, decision, scope switch). Skip it for routine log appends — bump on substance, not heartbeat. +- **Preserve `created:`.** It is the earliest known date the entity was tracked anywhere — never reset it to "today" when merging. +- **No `[[wikilinks]]` in frontmatter** — YAML parses them as nested lists and they break. Cross-references go in a `## Related` body section. `source_notes` holds plain relative-path strings (backward links to inputs only). + +--- + +## REST API operations + +Server `https://echoapi.alwisp.com`, bearer auth, root-addressed paths. + +| Operation | Method | Notes | +|-----------|--------|-------| +| Read file | `GET /vault/` | Raw markdown; `404` = absent | +| Read heading | `GET /vault//heading/

::

` | `::`-delimited, URL-encode spaces as `%20` | +| List dir | `GET /vault//` | Trailing slash; returns `{files, folders}` | +| Document map | `GET` + `Accept: application/vnd.olrapi.document-map+json` | Discover exact heading / block / frontmatter targets | +| Append | `POST /vault/` | Appends to end-of-file; creates if absent | +| Create / overwrite | `PUT /vault/` | Auto-creates intermediate directories | +| Patch section | `PATCH /vault/` | `Operation: append\|prepend\|replace`, `Target-Type: heading\|frontmatter\|block` | +| Search | `POST /search/simple/?query=` | Returns `[{filename, score, matches}]` | +| Delete | `DELETE /vault/` | Only on explicit operator request | + +**PATCH heading targets must be the full `::`-delimited path from the top-level heading** (e.g. `Operator Preferences::Fact / Pattern`) — a bare subheading name returns `400 invalid-target` (errorCode 40080). GET the document map first when unsure of the exact path, and copy the heading string verbatim. + +--- + +## Memory routing map + +| Situation | Vault path | Method | +|-----------|-----------|--------| +| Quick capture / unsorted | `inbox/captures/inbox.md` (date-prefixed line) | POST | +| Raw imported material | `inbox/imports/` | PUT | +| Operator preference / durable fact | `_agent/memory/semantic/operator-preferences.md` | PATCH | +| Other durable facts, patterns | `_agent/memory/semantic/.md` | PUT | +| Event record (what happened) | `_agent/memory/episodic/.md` | PUT | +| Short-lived, time-boxed state | `_agent/memory/working/.md` | PUT | +| Task-scoped context / focus | `_agent/context/current-context.md` | PATCH / PUT | +| Working-session log | `_agent/sessions/YYYY-MM-DD-HHMM-.md` | PUT | +| Long-running project state | `projects//.md` (folder and `status:` MUST agree) | PUT + PATCH | +| Standing area of responsibility (no end state) | `areas//.md` (`business`/`personal`/`learning`/`systems`) | PUT | +| Non-obvious decision (ADR) | `decisions/by-date/YYYY-MM-DD-.md` (mirror into a project's `## Key Decisions`; else skip) | PUT | +| Person context | `resources/people/.md` | PUT / PATCH | +| Concept / reference note | `resources/concepts/` or `resources/references/` | PUT | +| Meeting notes / call recap | `resources/meetings/YYYY-MM-DD-.md` | PUT | +| Skill / plugin capability entry | `_agent/skills/active/.md` (→ `archived/` when retired) | PUT | +| Daily activity / Agent Log | `journal/daily/YYYY-MM-DD.md` | POST / PATCH | +| Periodic review | `reviews/{weekly,monthly,quarterly,annual}/` | PUT | +| Bootstrap marker (plugin-owned) | `_agent/echo-vault.md` (`schema_version`, date) | GET / PUT | + +**Slug rules:** kebab-case, ASCII, ~40 chars max. Every file carries canonical YAML frontmatter. + +--- + +## Frontmatter conventions + +Every note opens with this block (fill what applies; leave the rest empty rather than guessing): + +```yaml +--- +type: # daily-note | project | area | concept | reference | person | + # meeting | decision | review | session-log | working-memory | + # episodic-memory | semantic-memory | context-bundle | skill | ... +status: # active | draft | done | archived | complete | incubating | on-hold +created: # YYYY-MM-DD — earliest known date the entity was tracked +updated: # YYYY-MM-DD — bumped on meaningful change +tags: [] +agent_written: # true on agent-generated notes +source_notes: [] # plain relative paths (backward links) — NEVER [[wikilinks]] +--- +``` + +`agent_written: true` + a populated `source_notes` is the key signal separating agent-managed content from human-authored content. + +### operator-preferences.md — Rules vs Observations + +- `## Fact / Pattern` — promoted, deduped, **timeless** rules (no date prefix). +- `## Observations` — **timestamped** raw observations (`- 2026-06-06: ...`); the default landing zone for new evidence. + +Observations are promoted into Fact / Pattern (dropping the date) once a pattern stabilizes, and trimmed to the last ~30 entries. + +--- + +## Bootstrap, repair & migration + +The plugin carries everything needed to stand up a vault in `scaffold/`; there is no dependency on any in-vault control doc. + +- **Probe** — GET `_agent/echo-vault.md`. `200` → bootstrapped (check `schema_version`); `404` → run a fresh bootstrap. +- **Fresh bootstrap** (idempotent, additive — never overwrites): create the folder tree (a one-line README seeds each leaf since Obsidian/git ignore empty dirs) → PUT the 8 templates to their mirrored vault paths → PUT the 3 anchor seeds only if absent (the operator-preferences seed is deliberately empty — no fabricated facts) → PUT the thin vault README → PUT the marker **last** (so the vault is only flagged "set up" once everything is in place) → write a first-run trace (daily note + session log). +- **Repair** — same steps, GET-probing each path and creating only what's missing; malformed files are flagged in the session log, never replaced. +- **Migrations** — when the marker's `schema_version` is older than the plugin's, apply each intervening migration then PATCH the marker. `0 → 1` removed the old in-vault control docs (`CLAUDE.md` / `BOOTSTRAP.md` / `STRUCTURE.md` / `index.md`) in favor of the plugin-as-source-of-truth model. + +--- + +## Skill triggers + +| Skill | Triggers | +|-------|----------| +| `echo-memory` | "remember that", "save to memory", "what do you know about me", "load my profile", "check my notes", "log this decision", "add to my inbox" — and proactively at the start of substantive conversations | + +--- + +## Safety rules (operating contract) + +- Do not fabricate facts, relationships, or prior decisions. +- Do not mass-restructure the vault unless explicitly asked. +- Do not delete notes unless deletion is explicitly requested and clearly safe. +- Never store secrets or API keys inside a vault note. +- Default to additive updates and explicit status changes over destructive edits. +- Write in **third person** about Jason ("Jason prefers X"). Do not cross-write with other vaults (Bryan's `goldbrain` is separate). + +If the API returns a connection error, timeout, or `502` (usually Obsidian / the REST plugin not running), tell Jason once that the vault is unreachable and proceed without memory — don't retry repeatedly. + +--- + +## Version history + +| Version | Highlights | +|---------|-----------| +| **0.3.0** | Source promoted from zip-only to a tracked tree (`echo-memory.plugin.src/`); `.plugin` becomes a build artifact. All 7 skill-improvement items applied: search-first before writes, resilient daily Agent Log, `created:` semantics, project lifecycle + folder↔status rule, canonical HHMM session filenames, read-most-recent-N sessions, `source_notes` defined as backward links. | +| **0.4.0** | Efficiency + robustness pass: parallel cold-start loading, idempotent POST (read-before-append), doc-map-before-first-PATCH, scoped `updated:` bump, Inbox Triage, Scope Switching, monthly Vault Health, Rules-vs-Observations split, formal deprecation of `decisions/by-project/`, heartbeat pointer. | +| **0.4.1** | Bugfix: daily-note Agent Log heading detection now greps raw markdown for `^## Agent Log` instead of the `::`-delimited doc-map JSON (which never matched and appended duplicate headings). Added Scope Switching cold-start test harness. | +| **0.5.0** | Self-bootstrap + control-logic-in-plugin. Plugin becomes the single source of truth: bundled `scaffold/` (8 templates, 3 anchor seeds, thin vault README, marker) bootstraps an empty vault with no external/local-path dependency. New `operating-contract.md` (principles + safety from the old in-vault `CLAUDE.md`); `bootstrap.md` rewritten as a portable bootstrap/repair/migrate manifest. Cold-start probe moved from `/vault/BOOTSTRAP.md` to `_agent/echo-vault.md` (carries `schema_version`). Live vault migrated to data-only. | +| **0.5.1** | Routing-doc consistency pass: decision-mirror heading unified to `## Key Decisions`; stale `Current status` PATCH examples corrected to `Status`; vault-layout inline project example refreshed to the real template. All 17 `projects/active/` notes normalized losslessly to the canonical template heading set; `android-mqtt-shell` moved to `incubating/` (was broken `status: upcoming` in active). Plugin repackaged (21 files). |