connect the graph, measure the retrieval it is for, and answer the review - #64
Merged
Merged
Conversation
Measured on this store 2026-09-13: the shared scope holds 246 entities with a median degree of 1, and 147 of them - 60% - appear in exactly one fact. The solo scope is the same at 63%. The shape is a few project hubs carrying 14 to 35 edges and a long tail of leaves that turn up once and are never mentioned again. That is a write-time outcome. Every episode arrives with its own freshly invented names, gets two new nodes, joins them to each other and stops. Resolution only intervenes when a new name nearly matches an old one, so an agent inventing a genuine near-synonym gets no signal at all. It matters now rather than eventually because multi-hop retrieval is the next version's whole argument, and a traversal across a median-degree-1 graph leaves a leaf, reaches a project hub and stops. No work on the read path fixes a graph with nothing to walk; the order has to be connectivity, then measurement, then traversal. So after an episode commits, find the facts already in scope nearest to what was just written and name the entities they connect, each with the fact that makes it relevant - evidence rather than an assertion the agent has no reason to trust. Similarity between fact texts, not between names. Name-against-name is what entity resolution scores, and calibration on this store puts it at AUC 0.666, 95% CI [0.421, 0.881], an interval containing chance. The bare fact text against stored vectors is the eval's `prose` shape, which measures R@1 0.949 and MRR 0.970 here - the best-performing channel in the system. Advisory and nothing else: it runs after the transaction, cannot write an edge, and is swallowed whole if it fails, because a graph is only worth reading if every edge in it was claimed by somebody.
A field no instruction mentions is a field no agent reads. This project has measured that twice: Claude Desktop answered three questions without once calling record_recall_save while the instruction block named three tools and not that one, and eight merged PRs produced zero writes with write_episode available and documented as proactive. So the reply's new field is described where an MCP client actually looks, in the same words it would need to act on: reuse the name next time you write about that subject, nothing is written from this, no reply is needed. It also says why, because "reuse names" without "otherwise the graph cannot be walked" is an instruction with no reason attached, and those are the ones that lose.
Every shape in the harness had a single edge as its gold answer, so all of it measured one-hop lookup. Multi-hop retrieval is the next version's stated differentiator - the README's "how did we end up here?" - and no change to it could have been shown to help or hurt, in a project whose whole discipline is that the evaluation overrules opinion. The new shape asks about two entities that no single fact connects but that both appear in facts about a third, so answering needs both edges. The labels stay free: the graph already knows which entities share a neighbour. Scoring changes with it. A case counts as answered at the rank of the LATER required fact, not the first: a run that returns one half at rank 1 and never the other has answered nothing, and a metric that rewarded it would make traversal look unnecessary. For every single-hop shape the rule is identical to the old one, because their required set holds one edge. Two bounds keep the sample honest. A hub incident to 35 facts offers 595 ordered pairs, so each middle entity contributes at most six cases - uncapped, one project node would supply most of the sample and the score would describe that node. And a pair a single fact already joins is excluded, because that question is single-hop and including it would flatter the result with cases that never needed a traversal. Baseline on this store, n=187: R@3 0.390, R@10 0.647, MRR 0.212, against 0.672 to 0.970 for the single-hop shapes. R@1 is 0 by construction and the shape note says so. That number is the thing traversal now has to beat.
The graph has been stored since migration 0001 and the read path has never touched it: two content channels fused, and an AGE structure nothing reads. This adds the third list - facts one hop from the facts a query already found, ranked by the seed they came from - behind graph_hops, defaulting to 0. It is off because the eval says so, not because it is unfinished. Scored over the new multihop shape, n=187, paired bootstrap against shipping: shipping R@3 0.374 R@10 0.647 MRR 0.211 + graph hop R@3 0.278 R@10 0.722 MRR 0.197 [-0.038, +0.010] ? The traversal is finding the right facts: R@10 0.722 is the best of any configuration in the table, shipping included. It is also ranking them worse - R@3 drops - and the MRR difference sits inside the noise. A channel that improves what is reachable while degrading what is read first is not a win, and fusing it as a peer of the content channels is the reason: its candidates are neighbours of an answer, not answers. So the mechanism ships measured and disabled, and the number it has to beat is written down. This also corrects a claim made while planning the work: that a median-degree-1 graph would make traversal return nothing. It does not follow - a hub with 35 degree-1 leaves joins every leaf to every other in two hops, and the 187 cases this shape generates are exactly those paths. What the degree distribution supports is that expansion runs through generic hubs and will be imprecise, which is what the R@3 column shows. One unrelated finding fell out of the same run: the static 0.15 floor, noise or worse on every one-hop shape, is a significant win on multihop (+0.0225, [+0.008, +0.037]). A second fact sits further from the query than the first, so admitting lower-similarity candidates helps exactly where it previously did not. Not acted on here - one shape is not grounds for changing a default.
This project's headline finding compared 0 of 16 memory files converted into facts against hooks firing 24 of 24. Those are different outcomes - one is end-to-end capture, the other is a trigger - and a reviewer of the paper drawn from this record rejected the comparison on exactly that ground. He was right, and the replacement number is worse than the claim it corrects. Measured on this store, both sides as knowledge actually stored: the Stop gate fired 7 times and 1 firing was followed by a fact from the same session. The gate is the strongest mechanism here - it returns a block decision, refuses to let a session end, and hands back the unwritten list as the next instruction. One session it held open had made 118 edits and still wrote nothing. So structural TRIGGERING is reliable and structural CAPTURE is not, because the last step hands a model a choice about what is worth remembering and there is no hook for that. It goes in `health` rather than a one-off script because the claim it corrects is load-bearing, and a number nobody runs is a number that drifts. It undercounts by construction - a fact written later under a different session id is not credited to the gate that asked for it, which is exactly what happened when 16 files were drained by hand ten days afterwards - so it prints as a floor.
Calibration reports 50% precision at the 0.92 auto-merge bar, over two reviewed
pairs. That is no evidence at all for an unattended graph mutation, on a store
that has already recorded three bad merges, and a reviewer of the paper said so.
What settled it was the cost. Every entity resolution this store has performed:
126 exact match
34 agent-confirmed fuzzy match
1 fuzzy match, similarity=0.927 <- the only silent merge, ever
One in 24 days. Turning it off buys about one confirmation prompt a month and
removes the only path that can join two entities without anyone seeing. The
pair still surfaces as ambiguous, so nothing is lost except the silence.
The constant stays, with the evidence beside it, so turning it back on is a
decision rather than an archaeology exercise. The bar for that is calibration
with enough positives for the AUC interval to clear 0.5, and precision at the
threshold resting on more than two pairs.
The trial fixture took the silent path to produce a non-exact merge; it now
does what a real agent does, which is answer the ambiguity.
The abstract number is "0 LLM calls per episode, $0.00", and it is true of the thing it measures: this server calls no model on the write path. It was being read as something larger - that recording a memory is free - which it is not. Extraction still happens, in the agent, and the 14.6ms is a server receiving facts somebody else already produced. So the section is "Server-side cost" now, and the paragraph under it names what is outside the boundary: the extraction itself, the tokens of the tool call, a correction turn, a capture prompt. The claim that survives is that this architecture adds no inference of its own, which is the claim worth making and is not the same as free. Same correction in the README, in both places it appeared.
Two things a reader of these tables could not know, and a reviewer of the paper did not. The intervals are a seeded paired bootstrap over per-case differences, 5000 resamples, 2.5th and 97.5th percentiles - not the per-shape standard error printed under each table. Both configurations score the same cases in the same order, so the variance that matters is that of the difference; bootstrap rather than a t-test because reciprocal ranks are 1, 1/2, 1/3 ... 0 with a spike at 0. The method was already there. Nobody could see it. And every query in this harness is derived from its own answer, so absolute scores measure how easy that derivation is. entity_single is worse than leaky: one fact is marked correct for a name that may appear in thirty, and the other twenty-nine relevant ones score as failures. The footer now says a number from this table does not belong in a claim about quality without hand-written questions and relevance labels permitting more than one right answer. The module docstring also records that one metric cannot judge every change. MMR was added for diversity; a single-answer metric is blind to diversity by construction. It is off because it costs ranking, and these tables say nothing about whether it delivered what it was added for.
…budget The server derives both tool identities from evidence it holds - the writer from the cited edge, the reader from the presented key - so "two different tools were involved" is verified and the caller cannot assert it. Everything before that is not: the agent chose which fact to cite and judged that it saved a re-explanation. The line now says so where the number is read. Separately, documenting related_entities pushed write_episode's description to 2244 characters against Claude Code's 2048 limit - the truncation is silent, a debug line and a cut string, and the last time it happened it removed the worked example. The opening paragraph is tightened and the new field described in three lines instead of seven: 1863 characters, 85 clear of the headroom guard that caught it.
The gate's conversion number was wrong twice within an hour of shipping, in both directions, and the cause was the same missing column. Counting only writes made it 1 of 8, and scored this session as a failure for doing exactly what the gate's own instruction asks: when a file's content is already recorded, mark it done rather than write it twice. A metric that punishes the behaviour its own instruction requests measures compliance with itself, not capture. Counting any closure after a firing made it 8 of 8, because pending_ingest recorded when a document was closed and nothing about by whom, so every historical firing matched something. Migration 0020 adds ingested_by_session, nullable and not backfilled - closures made before it have no recoverable author, and inferring one from a timestamp is how the second wrong number happened. Both close paths set it: the MCP tool takes the session_id the caller already passes write_episode, and the CLI takes --session, which the Stop gate now puts on the line it prints so the caller does not have to know to supply it. Back to 1 of 8 on the live store, and correct this time.
Three failures from the clean run, all consequences of this branch. The benchmark render test pinned "Zero by construction". The output now names the accounting boundary instead, so the test asks for that - and its docstring records why, because the phrasing is the fix for a real misreading rather than a style preference. The silent-merge test asserted a merge that no longer happens. Split in two: one that a near-match is offered for confirmation and creates no second node, and one that a confirmed match merges and appends the alias. The node-count assertion is the part worth keeping - a near-match must never mint a second node for one entity, which is the trial's duplicate bar - and it now holds by waiting for an answer rather than by guessing. The reattribution test gave every entity in its fixture the same vector, so a second episode's new entity scored 1.0 against an existing node and was absorbed silently. With that path gone it comes back ambiguous and the episode writes nothing, which is correct and left the test with no second session to reattribute. Postgres gets its own vector; the collision was never the subject.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Eleven commits in three groups: a write-path change aimed at graph connectivity, the measurement that makes the next phase falsifiable, and the corrections a peer reviewer of the paper drawn from this repo asked for.
The graph was barely a graph
Measured on the live store: the shared scope holds 246 entities with median degree 1, and 147 of them (60%) appear in exactly one fact. Solo is 63%. The shape is a few project hubs at degree 14–35 and a long tail of leaves that turn up once.
That is a write-time outcome. Every episode arrives with freshly invented names, gets two nodes, joins them to each other and stops; resolution only intervenes on a near name match, so an agent coining a genuine near-synonym gets no signal.
write_episodenow returnsrelated_entities— entities already in scope that appear in facts nearest what was just written, each with the fact that makes it relevant. Advisory only: it runs after the transaction, cannot write an edge, and is swallowed whole if it fails.It rides fact-text similarity, not name similarity, and that choice is measured rather than assumed: name-against-name calibrates at AUC 0.666, CI [0.421, 0.881] — an interval containing chance — while the bare fact text against stored vectors is the eval's
proseshape at R@1 0.949, MRR 0.970.Tried against the real store, it immediately surfaced two nodes that should be one (
AGE drops null properties on CREATEandAGE null property drop) — the fragmentation it exists to prevent.The eval could not see what the product is for
Every shape's gold answer was a single edge, so the whole harness measured one-hop lookup, while multi-hop retrieval is the next version's stated differentiator. No change to it could have been shown to help or hurt.
The new
multihopshape asks about two entities no single fact connects but which both appear in facts about a third. Labels stay free. A case counts as answered at the rank of the later required fact — a run returning one half at rank 1 and never the other has answered nothing.Two bounds keep the sample honest: at most six cases per middle entity (a degree-35 hub offers 595 ordered pairs), and pairs a single fact already joins are excluded as single-hop.
Traversal: built, measured, shipped off
One-hop expansion as a third RRF channel finds more — R@10 0.722 is the best figure in the table, shipping included — and ranks worse: R@3 drops and the MRR difference is noise. Its candidates are neighbours of an answer, not answers, so fusing it as a peer of the content channels is the wrong shape. It ships behind
graph_hops, default 0, with the number it has to beat recorded.This also corrects a claim made while planning the work — that a median-degree-1 graph makes traversal return nothing. It does not follow: a hub with 35 degree-1 leaves joins every leaf to every other in two hops, and the 187 cases are exactly those paths.
Unrelated finding from the same run: the static 0.15 floor, noise or worse on every one-hop shape, is a significant win on multihop. Not acted on — one shape is not grounds for changing a default.
The review
A reviewer raised seven points against the paper drawn from this record. Six are conceded and fixed here.
The capture claim was withdrawn. 0-of-16 (end-to-end capture) against 24-of-24 (a trigger) are different outcomes. Measured on one outcome: the Stop gate fired 8 times and 1 firing was followed by a fact. One gated session had made 118 edits and wrote nothing. Structural triggering is reliable; structural capture is not, because the last step still hands a model a choice. Now permanent in
echo-memory health.That number was itself wrong twice within an hour, in both directions, and migration 0020 is the fix — see below.
Unattended merging is off.
SILENT_MERGE = False. Calibration puts precision at the 0.92 bar at 50% over two pairs, and the audit log shows the path has fired once, ever, against 126 exact matches and 34 confirmed ones. It bought about one prompt a month and was the only way two entities could be joined with nobody watching.The zero names its boundary. "Server-side cost", with the excluded costs stated: extraction, the tool call's tokens, a correction turn. The claim that survives is that this architecture adds no inference of its own.
The harness states its limits. The footer now prints that intervals are a seeded paired bootstrap over per-case differences (the method existed; nobody could see it), and that every query is derived from its own answer so absolute scores are not quality. The docstring records that a single-answer metric is blind to the diversity MMR was added for.
The saves bar says what it establishes — provenance verified, benefit the agent's judgement.
Not conceded: the paired-difference method was already implemented in
compare(). That was a documentation gap.Migration 0020
The gate's conversion metric counted only new facts, and scored as a failure a session that did exactly what the gate's instruction asks — find the content already recorded, mark it done. Counting any closure instead gave 8 of 8, because
pending_ingestrecorded when a document was closed and nothing about by whom.ingested_by_session, nullable, not backfilled. Both close paths set it; the Stop gate puts--sessionon the line it prints so the caller need not know to supply it. Back to 1 of 8, correct this time.A metric that punishes the behaviour its own instruction requests measures compliance with itself, not capture.