docs(website): align mempalaceofficial.com with honest benchmarks
Part of #875. Bring the VitePress site into line with the new README and the reproducibility scorecard: drop category-error comparisons, drop retracted claims, retain only metrics and caveats that survive audit. website/index.md - New tagline matches README (local-first, verbatim, pluggable backend, 96.6% R@5 raw, zero API calls). - Replace the "MemPalace hybrid 100% / Supermemory ~99% / Mastra 94.87% / Mem0 ~85%" comparison table with a single honest table showing MemPalace's own retrieval-recall numbers (raw 96.6%, hybrid v4 held-out 98.4%). Add an explicit sentence explaining why we no longer publish a cross-system table on the landing page (retrieval recall vs QA accuracy are different metrics). - Soften the "ChromaDB-powered vector search" feature blurb to be backend-agnostic, since the retrieval layer is pluggable. website/reference/benchmarks.md - Full rewrite of the retrieval-recall tables. No more "100%" headline; honest held-out 98.4% R@5 replaces it. Added the model-agnostic rerank result (99.2% R@5 / 100% R@10 with minimax-m2.7 via Ollama) to show the pipeline is not Haiku-specific. - Drop the LoCoMo "Hybrid v5 + Sonnet rerank (top-50) 100%" row. With per-conversation session counts of 19-32 and top_k=50, the retrieval stage returns every session by construction — the number measures an LLM's reading comprehension, not retrieval. - Drop the cross-system comparison tables. Link out to each project's own research page (Mastra, Mem0, Supermemory) for their published numbers and metric definitions. - Rewrite reproduction commands to use the correct repository and demonstrate the new --llm-backend ollama flag. website/concepts/the-palace.md - Remove the "+34%" row / paragraph. Wing/room filtering is standard metadata filtering in the vector store, not a novel retrieval mechanism — the April-7 note already retracted that framing; this finishes the retraction on the website where it had remained. website/guide/searching.md - Same treatment for "34% retrieval improvement". Reframe as operational scoping, not a novel boost. website/reference/contributing.md - Update the "palace structure matters" bullet to reflect the same framing: scoping-not-magic. website/concepts/knowledge-graph.md - Replace the MemPalace-vs-Zep feature matrix with a short "related work" note that links to Zep's own documentation for authoritative details on their deployment model. Avoids claims we cannot verify at source.
This commit is contained in:
@@ -80,12 +80,11 @@ The knowledge graph uses SQLite with two tables:
|
||||
|
||||
Database location: `~/.mempalace/knowledge_graph.sqlite3`
|
||||
|
||||
## Comparison
|
||||
## Related Work
|
||||
|
||||
| Feature | MemPalace | Zep (Graphiti) |
|
||||
|---------|-----------|----------------|
|
||||
| Storage | SQLite (local) | Neo4j (cloud) |
|
||||
| Cost | Free | $25/mo+ |
|
||||
| Temporal validity | Yes | Yes |
|
||||
| Self-hosted | Always | Enterprise only |
|
||||
| Privacy | Everything local | SOC 2, HIPAA |
|
||||
Temporal entity-relationship graphs are a familiar pattern — Zep's
|
||||
Graphiti, for example, also exposes a bi-temporal model. MemPalace's
|
||||
knowledge graph is local-first (SQLite, everything on disk) and free;
|
||||
Zep is a managed service backed by Neo4j with its own pricing, SLAs,
|
||||
and compliance surface. See Zep's own [documentation](https://www.getzep.com/)
|
||||
for authoritative details on their deployment model.
|
||||
|
||||
@@ -92,16 +92,9 @@ The original stored text chunks. This is the primary retrieval layer used by the
|
||||
|
||||
## Why Structure Matters
|
||||
|
||||
Tested on 22,000+ real conversation memories:
|
||||
Wing and room identifiers become metadata filters at query time. Narrowing a search to a specific wing (or wing + room) means the vector store only scores candidates inside that scope, which is useful when you have many unrelated projects or people filed in the same palace.
|
||||
|
||||
| Search scope | R@10 | Improvement |
|
||||
|-------------|------|-------------|
|
||||
| All closets | 60.9% | baseline |
|
||||
| Within wing | 73.1% | +12% |
|
||||
| Wing + hall | 84.8% | +24% |
|
||||
| Wing + room | 94.8% | +34% |
|
||||
|
||||
The practical point is that structure improves retrieval. In the project benchmarks, narrowing the search scope by wing and room outperformed searching the entire corpus at once.
|
||||
This is standard metadata filtering in the underlying vector store, not a novel retrieval mechanism. The useful property here is operational — clear scoping rules that a human or an agent can apply predictably — not a magic retrieval boost.
|
||||
|
||||
## Navigation
|
||||
|
||||
|
||||
Reference in New Issue
Block a user