Igor Lins e Silva
a4868a3589
The miner upserted one drawer per ChromaDB call, paying tokenizer + ONNX session setup per chunk. The embedding device was CPU-only because no EmbeddingFunction was ever wired through the backend. Two changes, each a speedup in its own right; stacked they give ~10x end-to-end on a medium corpus (20 files, 568 drawers): 1. Batched upsert. `process_file` and `_file_chunks_locked` now collect all chunks of a file into a single `collection.upsert(...)` so the embedding model runs one forward pass per file instead of N. 2. Hardware-accelerated embedding function. New `mempalace/embedding.py` wraps `ONNXMiniLM_L6_V2` with configurable `preferred_providers`. `MEMPALACE_EMBEDDING_DEVICE` (or `embedding_device` in config.json) selects auto / cpu / cuda / coreml / dml. Unavailable accelerators log a warning and fall back to CPU. The factory subclasses `ONNXMiniLM_L6_V2` and spoofs its `name()` to `"default"` so the persisted EF identity matches existing palaces created with ChromaDB's bare `DefaultEmbeddingFunction` -- same model, same 384-dim vectors, no rebuild needed when turning GPU on. `ChromaBackend.get_collection` / `create_collection` now pass the resolved EF on every call so miner writes and searcher reads agree. Benchmarks (i9-12900KF + RTX 3090, medium scenario, 568 drawers): per-chunk + CPU 19.77s · 29 drw/s (baseline) batched + CPU 8.07s · 70 drw/s (2.4x) batched + CUDA 2.15s · 264 drw/s (9.2x) Reproducible via `benchmarks/mine_bench.py`. Install paths: pip install mempalace[gpu] # NVIDIA CUDA pip install mempalace[dml] # DirectML (Windows) pip install mempalace[coreml] # macOS Neural Engine Mine header now prints `Device: cpu|cuda|...` so users can confirm the accelerator engaged.
MemPalace Benchmarks — Reproduction Guide
Run the exact same benchmarks we report. Clone, install, run.
Setup
git clone https://github.com/MemPalace/mempalace.git
cd mempalace
pip install -e ".[dev]"
Benchmark 1: LongMemEval (500 questions)
Tests retrieval across ~53 conversation sessions per question. The standard benchmark for AI memory.
# Download data
mkdir -p /tmp/longmemeval-data
curl -fsSL -o /tmp/longmemeval-data/longmemeval_s_cleaned.json \
https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json
# Run (raw mode — our headline 96.6% result)
python benchmarks/longmemeval_bench.py /tmp/longmemeval-data/longmemeval_s_cleaned.json
# Run with AAAK compression (84.2%)
python benchmarks/longmemeval_bench.py /tmp/longmemeval-data/longmemeval_s_cleaned.json --mode aaak
# Run with room-based boosting (89.4%)
python benchmarks/longmemeval_bench.py /tmp/longmemeval-data/longmemeval_s_cleaned.json --mode rooms
# Quick test on 20 questions first
python benchmarks/longmemeval_bench.py /tmp/longmemeval-data/longmemeval_s_cleaned.json --limit 20
# Turn-level granularity
python benchmarks/longmemeval_bench.py /tmp/longmemeval-data/longmemeval_s_cleaned.json --granularity turn
Expected output (raw mode, full 500):
Recall@5: 0.966
Recall@10: 0.982
NDCG@10: 0.889
Time: ~5 minutes on Apple Silicon
Benchmark 2: LoCoMo (1,986 QA pairs)
Tests multi-hop reasoning across 10 long conversations (19-32 sessions each, 400-600 dialog turns).
# Clone LoCoMo
git clone https://github.com/snap-research/locomo.git /tmp/locomo
# Run (session granularity — our 60.3% result)
python benchmarks/locomo_bench.py /tmp/locomo/data/locomo10.json --granularity session
# Dialog granularity (harder — 48.0%)
python benchmarks/locomo_bench.py /tmp/locomo/data/locomo10.json --granularity dialog
# Higher top-k (77.8% at top-50)
python benchmarks/locomo_bench.py /tmp/locomo/data/locomo10.json --top-k 50
# Quick test on 1 conversation
python benchmarks/locomo_bench.py /tmp/locomo/data/locomo10.json --limit 1
Expected output (session, top-10, full 10 conversations):
Avg Recall: 0.603
Temporal: 0.692
Time: ~2 minutes
Benchmark 3: ConvoMem (Salesforce, 75K+ QA pairs)
Tests six categories of conversational memory. Downloads from HuggingFace automatically.
# Run all categories, 50 items each (our 92.9% result)
python benchmarks/convomem_bench.py --category all --limit 50
# Single category
python benchmarks/convomem_bench.py --category user_evidence --limit 100
# Quick test
python benchmarks/convomem_bench.py --category user_evidence --limit 10
Categories available: user_evidence, assistant_facts_evidence, changing_evidence, abstention_evidence, preference_evidence, implicit_connection_evidence
Expected output (all categories, 50 each):
Avg Recall: 0.929
Assistant Facts: 1.000
User Facts: 0.980
Time: ~2 minutes
What Each Benchmark Tests
| Benchmark | What it measures | Why it matters |
|---|---|---|
| LongMemEval | Can you find a fact buried in 53 sessions? | Tests basic retrieval quality — the "needle in a haystack" |
| LoCoMo | Can you connect facts across conversations over weeks? | Tests multi-hop reasoning and temporal understanding |
| ConvoMem | Does your memory system work at scale? | Tests all memory types: facts, preferences, changes, abstention |
Results Files
Raw results are in benchmarks/results_*.jsonl and benchmarks/results_*.json. Each file contains every question, every retrieved document, and every score — fully auditable.
Requirements
- Python 3.9+
chromadb(the only dependency)- ~300MB disk for LongMemEval data
- ~5 minutes for each full benchmark run
- No API key. No internet during benchmark (after data download). No GPU.
Next Benchmarks (Planned)
- Scale testing — ConvoMem at 50/100/300 conversations per item
- Hybrid AAAK — search raw text, deliver AAAK-compressed results
- End-to-end QA — retrieve + generate answer + measure F1 (needs LLM API key)