Every memory product says some version of “it really knows you.” That sentence is either a measurement or a vibe. We decided ours had to be a measurement, so we ran the real extraction pipeline — the exact code that ships — over real exported histories from ChatGPT, DeepSeek and Grok, and then graded the output two ways.
Two checks, because they fail differently. Verbatim: does the cited quote literally appear in the source? That’s code, no model, no judgment. Grounded: does the quote actually support the claim built on it? That needs a judge, so the judge works blind — it sees a claim and a source and never knows which answer helps us.
| Source | Conversations | Claims | Grounded | Quote verbatim |
|---|---|---|---|---|
| ChatGPT | 8 | 148 | 98.6% | 95.3% |
| DeepSeek | 8 | 57 | 94.7% | 96.5% |
| Grok | 6 | 63 | 100.0% | 95.2% |
| Combined | 22 | 268 | 98.1% | 95.5% |
The run that went against us
The first run didn’t say 98.1%. It said 68% — alarming enough to stop everything. The cause turned out to be the judge, not the engine: it was reading truncated sources and declaring real quotes fabricated. We fixed the harness, re-ran, and kept the broken number in the published record, because a benchmark that can’t be wrong can’t be trusted — and neither can a company that only publishes the runs it likes.
The claim we gave up
We also wanted to say “Lethe saves you tokens — you stop re-explaining.” The measurement said no: our context injection costs ~543 tokens against a 29-token median conversation opening; only 5 of 44 real conversations broke even. So that claim is dead, and what we say instead is what’s true — you stop performing the introduction, and answers start from your actual situation.
Precision without recall would be half a picture, so the other half is published too — it’s less flattering, and it has its own essay: What a careful reader finds that extraction can’t.