The failures that never throw
Six defects that never raised an error. All of them used one empty array for both "nothing there" and "could not fetch"; four were caught before deploy.
The report came in like this.
When I talk to the agent first thing in the morning it knows none of my preferences. From the second message on it's fine.
No error logs, no 5xx. The response was a 200 with an empty array where the memories should have been. Calling it repeatedly during the day never reproduced anything.
Once I found the cause, I realised there were five more defects of exactly the same kind.
Idle overnight, then the first request
The vector store's connection pool had no check before handing a connection out. Leave it idle for a few hours and the server drops the connection; the pool doesn't know, and hands over a dead one. The first request of the day takes it and fails.
01First turn of the morning. Nothing has touched this path since last night.
You can't drop a conversation because memory lookup failed, so swallowing the failure was the right design. The problem was that the swallowed failure was reported nowhere, which left an empty array and "a user with no memories" indistinguishable.
The fix was two lines: a pre-borrow check on the pool, and a signal for the swallowed failure that is distinct from an empty result.
Rows without the field never match the condition
To exclude the current conversation from search results I added a $not condition. And then memories the model had written itself stopped appearing in search entirely.
In SQL, NOT NULL evaluates to NULL. And the condition raises nothing, so the query succeeds and the result set just quietly shrinks.
Wrapped it in COALESCE(NOT (…), TRUE), and made the field non-optional in the first place by requiring a category on automatically ingested rows too.
Conversations after midnight belong to yesterday
Memories carry dates. Storing "half day off next Tuesday" requires a reference date.
The vendored code computed that reference in UTC.
Someone suggested setting the pod timezone to Asia/Seoul. We didn't, because that setting would have to be verified on every deploy, and if it's missed anywhere, the same bug comes back silently.
The reference date is pinned to UTC+9 in code instead. Korea has had no daylight saving since 1988, so the value is always correct.
The other three
There were three more of the same character, all caught before deployment.
The LIMIT without an ordering is the nastiest. Paging through, some items appear twice and others never appear at all — and querying once looks completely fine.
What they share
Laid side by side, all six collapse into one thing.
None of them throw. The dead connection became a swallowed failure, NOT NULL quietly shrank the result set, the UTC reference produced a plausible date, the full scan was merely slow, the string timestamp only got the order wrong, and LIMIT without a sort answered differently each time. And all of them returned a 200 with a plausible value.
What catching them early was worth
Four of the six were caught while still unreleased, by reading the code before any incident.
After release each would have needed a data migration. String timestamps mean converting a ledger that already exists, and for the UTC reference date there's no way to restore relative dates already stored. A memory that stored "tomorrow" a day off isn't recovered by fixing the timezone later.
Four of the six used one empty array for both "none" and "couldn't fetch". I suspect there are more of those in the code, and the proper fix is probably to go through every place that returns an empty array and give the two cases different types.
End