Four days ago we published a number: 98.1% of what Lethe stores is grounded in the line it cites. A fair reader should ask the question we then had to ask ourselves — is that good? A number without a denominator is a vibe. So we took the same 22 real conversations from that benchmark and fed them through the two best-known open-source memory pipelines: Mem0, and Graphiti — the engine inside Zep.
Everything that could be held equal, was. All three pipelines ran the identical extraction model, so the differences below are pipeline design, not model size. Every stored claim from every system was graded by one judge that sees only the source conversation and the claim — never which system produced it, never a confidence score, never a quote. Same retry budget for everyone. Same conversations, frozen before any arm ran.
| One judge for everyone | Lethe (fast) | Lethe (shipped) | Mem0 | Zep (Graphiti) |
|---|---|---|---|---|
| Claims stored | 211 | 218 | 214 | 651 |
| Grounded in the source | 100.0% | 93.6% | 98.6% | 82.8% |
| Cites the exact line | yes — 99.5% verbatim | yes — 97.7% | no | no |
| Conversations stored as nothing | 0 | 0 | 3 | 2 |
| Median seconds per conversation | 51.5 | 156.7 | 6.0 | 16.3 |
What we won
Every one of Lethe’s 211 claims survived the blind judge, and it is the only system in the table that can show you the line each claim came from — which is the entire design argument. Zep’s 17.2% fabrication rate is not judge pedantry; it is hypotheticals stored as history. A plan that suggested customers might buy a missing item became the stored facts “customers bought pant” and “customers bought jewellery.” Nobody bought anything. There is no line in the source you could cite for those claims — and that is exactly why we require one.
What we lost
Two columns, plainly. Mem0 is 8.6× faster — six seconds to our fifty-one — and against the reasoning tier we actually ship, 26× faster and more grounded: the shipped tier scored 93.6% on this judge, below Mem0’s 98.6%. And Zep’s volume is real: throw away every fabricated claim and Graphiti still banked about 2.5× more grounded material per conversation than we did. Precision is easy to game by saying less. Zep says a lot, and some of what it says and we don’t is signal we missed.
The bug it found in us — again
Our shipped tier’s 14 failed claims are not new failures. Nine are bare values — “60 days”, “10-12k”, two claims that are literally the word “none” — a schema shape where the meaning lives in a field the claim text doesn’t carry. The 2026-08-20 benchmark flagged this exact shape. It is still unfixed, and it just cost us a headline row against a competitor. We did not soften the grading to compensate: the rubric was frozen before results, and a rubric edited after seeing the numbers is a marketing exercise.
One more uncomfortable finding: our fast tier scored 100% partly because it never emits the fragile observation types the reasoning tier uses. Whether those richer types earn their failure rate — and their three-times latency — is now an open product question that this run forces, which is what benchmarks are for.
Then we tested whether the volume gap was self-imposed
Zep’s one real win was volume — and our prompt limits volume on purpose: it literally says “prefer fewer, non-overlapping observations.” So we ran the benchmark once more with only that instruction relaxed — same model, same pipeline, dedup and quote gate untouched, just permission to pull several distinct facts out of one sentence. The result: 62% more claims (354, at 16 per conversation), precision unchanged — 94.1% grounded against the shipped prompt’s 93.6% — at identical latency, because a reasoning model’s time goes to thinking, not to writing more lines. The volume cap was buying nothing. The costs, measured and small: verbatim quoting slipped from 97.7% to 94.9%, and every new failure was the same known bare-value bug, at a slightly lower rate. Fix that one bug and roughly 98% grounded at 15 claims per conversation is in reach — a projection, so it gets measured next, not claimed now. The shipped prompt is unchanged until then; this essay would rather report an experiment than quietly re-run until the table looked better.
Useful is a harder bar than true
A memory can be perfectly grounded and still be junk — nobody’s assistant answers better for knowing “bubble gum is sold at the vending machine.” So a third blind pass asked one more question of all 1,648 claims, rubric frozen first: would an assistant that knows this answer this person better someday? The bar reshuffles the race. Mem0 stores the cleanest memory of anything we tested — 91.6% of its claims are useful and true, beating the tier we ship on rate and on useful claims per conversation. Zep’s mountain mostly deflates: less than half of its 651 claims clear the bar — 175 are generic facts about the world, 113 are bare fragments — so its volume lead over our exhaustive arm shrinks to 12.6 against 11.6 per conversation. And our exhaustive arm passed the test that mattered: its extra volume held the same signal density as the shipped prompt, so the +57% useful claims are substance, not padding.
Two honest limits on this axis. Claims are graded one at a time, and some claims only become valuable when merged — ten unremarkable data points that add up to one real pattern across conversations. That value is invisible here by construction, which is why our extraction deliberately stores atoms and derives patterns in a separate pass over the whole history. And “useful” here means always injectable — but grounded claims have different service lives, not different worth: a meeting time is what reminders exist to consume, useful at the right moment rather than never, and a fact the writer researched is the decision landscape they were standing in. Nothing gets discarded for grading poorly on this axis. The table is a floor, not a ceiling — but the floor still separates material that serves the writer now, later, or merged from fabrications and empty values, which serve no one at any time.
We audited the judge, too
A blind judge is still a model, so every failed verdict in the run — all 129 across the three systems — was audited by hand against its source. Ours and Mem0’s held. Eleven of Zep’s did not: claims like “considers loom video” were in the source all along (“i wanted to decide fake cart test or loom video”), missed by a judge reading twelve thousand characters. Audit-adjusted, Zep’s floor rises to about 85%. The ordering doesn’t change, and the published number stays the measured one — but a scoreboard that audits its opponents’ losses and not its referee isn’t honest either.
The caveats that keep the table honest
We ran Mem0 and Zep as published open source, on the model we had keys for — not their hosted platforms, which likely run stronger defaults. Both were configured fairly (their real pipelines, local embeddings, same retries), but both were surely tuned by their authors against other models first. The judge shares a model family with every arm, blindly and identically. All 22 conversations are one person’s. And 100% on 211 claims is a ceiling on this sample, not a guarantee — at this size, the run can only certify the true error rate is below roughly 1.5%. The full method, configs, versions, and every failed claim are in the repo, reproducible with four commands.
By evening, the fixes had shipped — and been re-measured
The bare-value bug and the question-into-goal habit were fixed the same day: in the prompt, and at a normalization gate that drops placeholder claims loudly. Then the whole benchmark ran again. The shipped prompt now grounds at 98.8% — past Mem0’s 98.6% — at 18.8 claims per conversation, double the morning’s volume. And a third arm pushed the volume question to its end: three focus-lensed passes over the same conversation, run in parallel and unioned, stored 1,236 claims across the 22 conversations — 1.9× Zep’s entire count — still 98.5% grounded, still citing every line.
Then we applied to ourselves the skepticism Zep got: paraphrase dedup, one frozen rule, every system. The union arm loses 21% of its count to cross-lens paraphrases; Zep loses 35.9% of its; Mem0 loses exactly nothing, which is worth respecting — its reconcile step does real work. Deduped and filtered to claims that are distinct, true, and worth keeping, the union arm holds 646 — 29 per conversation, 3.2× Zep, matching Zep’s raw total with claims that all carry receipts. The costs stay in the record too: triple the token bill, a usefulness rate that dips as the net widens, and Mem0 still ingesting twenty times faster than anything we run.
The last arm, and the metric we chose
We then chased Mem0’s 91.6% usefulness rate directly — six more rules against our residual failures, plus a code repair for stitched quotes. The precision axes hit their best numbers yet: 99.1% grounded, 99.3% verbatim, 99.3% strict, 0.9% fabricated. The rate barely moved — and the data says why: our rate isour mix. The claims the rubric scores down aren’t junk anymore; they’re dream narrative, dated one-offs, the decision landscape — material with a time- or synthesis-scoped service life that Mem0 simply never stores. Their 91.6% is the rate of a narrow net. So the metric we now hold ourselves to is the absolute: useful-and-true claims per conversation — 13.7 to their 8.9 — everything a narrow net keeps, plus what it throws away, at higher precision. The rate stays in every table; a retired target is not a hidden one. One last arm made the point conclusive: we had the extractor declare its own inject-always core, and that self-declared core graded 70.7% — barely above its average. Proven three ways, the rate is the mix; the arm that chased it measured worse where it counts, so it isn’t what ships, and the negative result is published with the rest.
We could have published only the first row. The other rows — the losses, the judge’s own errors, the bug that cost us a headline and the evening it took to win it back — are how you know the first one is real.