Setting the 0.85 similarity threshold from sentence pairs
I started at 0.9, then measured real sentence pairs through the production embedding model and landed on 0.85. The two directions of failure cost different amounts, so the slack went to one side; on the search path I dropped the threshold entirely.
The last post was about the memory service attached to an agent. This one is about a single number buried inside it.
A user says "I'm moving in December" and the model writes that down. But last month it already stored "moving in November". Write a new one, overwrite the old one, or keep both?
The question is whether the two sentences are the same fact, and that's judged by embedding similarity. So from what number up should they count as the same?
Where the number sits
Hover a node to reveal its connections.
Two thresholds are needed: one for merging duplicates within a category, another for overwriting an older, automatically collected memory. The second is the more dangerous one, because getting it wrong erases a different fact.
What happens if you guess
We started at 0.9, because a high bar seemed like the safe one.
Production disagreed immediately. A user corrected their moving date to December, and the stale "November" memory kept coming back as the top search result. The correction hadn't failed to save; both were alive, and the stale one simply retrieved better.
I'd assumed a higher threshold was safer, but the higher it sat, the more often a correction failed to overwrite the stale memory and both stayed.
Checking it with sentence pairs
We collected sentence pairs and scored them with the same embedding model production uses. The pairs fall into four groups, because what we want differs by group.
- Same fact — different wording. Must overwrite
- Correction — the value changed. Must overwrite, or the failure above happens
- Added detail — the same fact with reasoning attached. Overwriting loses the elaboration
- Different fact — same category, different content. Overwriting destroys someone else's memory
The first two belong above the line and the last two below it, so the job is to lay the four distributions on one axis and find where to cut.
Press 0.80 and two of the four added-detail pairs get overwritten. Push to 0.90 and nothing is wrongly overwritten, but six of twenty same-fact pairs are missed. Only at 0.85 are both protected groups intact.
The margins on either side of 0.85
I picked 0.85 because the two directions of failure don't cost the same.
If an added detail gets overwritten, you lose the why but keep the what, and history can restore it. If a different fact gets overwritten, the fact itself is gone, and nobody knows it's gone.
So the margin went to the dangerous side, and that's where the gap between 0.22 and 0.006 comes from.
The same principle set the merge threshold, measured separately over 28 pairs.
| Group | Meaning | min | max | mean |
|---|---|---|---|---|
| Near duplicate | same fact, different wording — merge | 0.8217 | 0.9706 | 0.8885 |
| Legitimate update | same subject, changed value | 0.7508 | 0.9543 | 0.8612 |
| Different fact | same category only — never merge | 0.3225 | 0.7113 | 0.5291 |
The two distributions do not overlap. A gap of 0.1104. Anywhere inside it would work, but the margin went to the do-not-merge side, so 0.80 — 0.089 below, 0.022 above.
Where similarity stops working
Measuring surfaced a region no threshold reaches.
Same kind of correction, but it's caught when the sentence frame is preserved and missed when it isn't. Lowering the threshold can't fix it either, since going lower starts overwriting genuinely different facts too.
I don't think this is a question similarity can answer at all. My view is that the model has to name the memory it's replacing, explicitly.
On the read path we dropped thresholds entirely
Thresholds worked on the write path, so the obvious move was to add one on the read path too, cutting low-scoring search results. The usual thing.
The measurement said otherwise.
Cutting made it worse. The cause: score scale differs by query type. A topical query's top hit scored 0.397; a personal-fact query's top hit scored 0.092. Put them on one axis and the personal-fact side loses its correct answers first.
The same 0.3 was a top hit for one query and noise for another. On the write path two sentences are compared directly, so the score itself means something; in search, the range the scores fall in shifts from query to query.
So search has no absolute threshold, only ordering, and narrowing the results is done with conditions instead of scores.
+20%p. The scope filter was there as a security boundary; it turned out to be a retrieval-quality device too. Removing what is definitely wrong beats cutting by score.
How far this measurement goes
It isn't a significance test; it's a decision-support measurement of whether the empirical distributions of real pairs separate. Embeddings are deterministic, so the same pair always scores the same and repeating the measurement means nothing. The one uncertainty left is whether this sample represents the population.
One of the protected groups is thin. Added-detail has only four pairs, because that shape of memory is rare in the corpus and a second mining pass couldn't grow it. So the 0.006 margin can't be tightened by collecting more samples; it has to be corrected against the score distribution in production logs.
The domain is narrow too. The pairs come from conversations with three work personas, so I don't know whether other domains would give the same values.
Now, when someone asks where 0.85 came from, there's a table to show them. The reframed corrections probably need something other than a threshold. Before that, I plan to count how often production scores land near 0.85; if they do often, the threshold has to go up.
End