Long-term memory for LLM companions, and the harness that measures whether it actually remembers.
Extracted from a voice companion I built in 2026. This is the memory subsystem and its evaluation, separated from the product it shipped in.
uv sync # or: pip install -e '.[dev]'
uv run pytest # 114 tests, no network, no keys
uv run python -m friendly_memory.eval.harness # persona benchmark (slow, real LLM)Most agent memory silently truncates when a context budget fills. This does the opposite: an over-cap write is a hard error that returns the current block contents and asks the model to consolidate first.
try:
block.add("Started a new job at a design studio")
except BlockFullError as e:
# e carries the full block back to the model:
# "merge or drop something, then retry"Triage happens exactly when new information competes with old for space, which is the only moment you have enough context to decide what matters. Silent truncation makes that decision by accident, at the wrong end of the buffer.
Two capped blocks are injected every turn with a live capacity gauge
([67% — 1,474/2,200 chars]): a user block (who they are) and a self
block (the relationship). Everything else lives in an unbounded episodic store
searched on demand — capped blocks are for facts you should never have to search
for.
The mechanic is borrowed from Hermes; the design notes credit what was copied and what was changed.
| Path | What's in it |
|---|---|
memory/blocks.py |
Capped blocks, entry-granular edits, BlockFullError |
memory/store.py |
Episodic store interface, day summaries |
memory/backends/ |
SQLite FTS5 and Postgres implementations behind one protocol, with a parity test suite |
companion.py |
Turn loop: prompt assembly, post-turn fact extraction, consolidation retry |
brain.py |
Model adapter — swap Claude / Groq / Gemini without touching memory |
eval/ |
Persona time-lapse harness, judge ensemble, variant comparison |
variants/ |
Alternative memory strategies, benchmarked against the default |
Claiming an agent "remembers" is easy. The harness exists because I didn't believe the claim without a number.
It simulates personas over days of conversation — with scheduled fact updates and deliberate contradictions — then quizzes the companion from the persona's ground-truth plan rather than the transcript, so it can't score by echoing.
Three things it does that a naive eval doesn't:
- Embellishment screening. An answer passes only if it's correct and asserts no specific detail absent from ground truth. Confidently inventing a detail is a failure, not a near-miss.
- Judge ensemble. Two model families judge independently; agreement stands, disagreement escalates to a stricter tiebreak, and the per-run disagreement rate is printed. That number is the measurement noise your target has to clear — otherwise you're tuning against judge variance.
- Delayed recall. The delayed subset is re-asked ~10 simulated days later. That line is the forgetting curve.
variants/ exists so strategy changes are comparable rather than argued about:
implement a Companion subclass, run it through the same harness, read the
delta.
memory/backends/ puts SQLite FTS5 and Postgres behind one protocol, with a
shared parity suite asserting identical behaviour across both — so local
development and deployment can't quietly diverge. Postgres tests skip unless
FRIENDLY_MEMORY_TEST_PG_DSN is set.
114 passing, no network and no API keys required. The 15 skips are the Postgres parity suite, which needs a live database.
The parts under test are the parts that are hard to reason about: cap enforcement and consolidation depth, duplicate rejection, day-summary boundaries, backend parity, and prompt assembly ordering.