The hard exam

Every memory product publishes a number. Kit has one too: on a sealed quiz in July, full Kit answered thirty of thirty-two. It is a true number and it is worth very little, because a plain search over the same notes answered twenty-six, and the reason was the quiz, not the contestants. Two hundred and eighty-six tidy cards, and almost every question answerable from one card that shared its words. A search box is very good at that.

This note is about the exam we wrote to be hard, what it found, and what it costs to say the number out loud.

Why a harder exam

A real memory is not tidy. Facts get corrected quietly, months later, in different words. The same thing is said three ways. Two notes flatly contradict each other and nothing settles it. Names nearly collide. And the question you ask in October uses none of the words you wrote down in March.

So we added nine hundred and sixty cards to the July ones, one thousand two hundred and forty-six in all across five story worlds, and salted them with exactly those things. A custodian who never speaks to Kit’s builder wrote a hundred and twenty questions in seven kinds, sealed them, and published the checksums before anyone answered. Thirty of the questions have no answer in the notes, or an answer in a case file the contestant is not allowed to read; the right response to those is to say so.

Five contestants sit it: nothing at all, a plain hybrid search over the same cards, Kit’s search on its own, the full Kit with its cautionary gates on, and Graphiti, an open-source knowledge-graph memory, read the way its documentation says to read it. Every answer gets three marks. A strict key looks for exact phrases. A blind judge, a separate model that never learns which contestant it is marking, reads for meaning. And a third mark asks the simplest question: was the right note even found.

What it found first

Kit as it was found the right note seventy-seven times out of a hundred and twenty. The plain search box found it ninety-five times. That was the moment the new exam earned its keep, and it took two days to work out why.

The first fault was a word rule. Kit’s keyword search required every word in the question to appear in the note. Ask a natural question, with a few words the note never used, and that whole side of the search returned nothing. The fix was not to loosen it, which we tried and measured and reverted, but to weight every word by how rare it is across the whole store. A surname counts for a lot. The word the counts for nothing. Found-the-note went from seventy-seven to ninety-six, past the search box for the first time.

The second fault made things worse before it made them better. Kit has a conflict detector: when a new note arrives it looks for older notes that disagree with it. On this corpus it found six. The corpus has hundreds of planted disagreements. It was only comparing notes that used the same wording, so a quiet correction in different words walked straight past. We widened the net to a note’s nearest neighbours on subject, and it found three hundred and seventy-eight.

Then Kit’s resolver went to work on them, merging the pairs it could, and the score went down. When it merged two notes into one, the merged note lost its labels. It no longer said which case file it belonged to or which cards it came from. Under the exam’s rules a question about one case file may only look at that case file’s notes, so the merged notes were invisible, and the originals had been retired into them. The detector found the conflicts and the resolver hid the answers. A merged note now keeps every label of both parents, and picks the newer of two notes by when the thing happened rather than when it was written down.

The rival

Graphiti had only ever seen the July cards. To sit the new exam it had to read the nine hundred and sixty new ones, and reading is where a graph memory spends its money: every card goes through a large model to pull out people, places and facts. Peter set a cap of a hundred dollars at list price. It took three attempts, one stopped by a one-second timeout in the graph database, one by the provider account running dry, and the third wrote every remaining card without a failure. Fifty-eight dollars in all.

Kit read the same one thousand two hundred and forty-six cards for two dollars and nineteen cents, and all of that was the conflict judge. Per card, the rival’s write path is about twenty-nine times dearer. That is not a rounding difference; it is a difference in what the two products do when a note arrives, and it was unmeasured on both sides until this week.

On the exam Graphiti found the right note forty-eight times out of a hundred and twenty. Most of its score is the thirty questions where refusing is the right answer; it got twenty-six of those. Of the ninety with a factual answer it found the right card for eighteen. We checked every graph against the sealed cards, and every card is in there. Its own search does not surface the ones the question needs.

Where it stands

The column we quote is the ruled one: the strict key’s verdict, corrected only on the rows where the key and the judge disagreed, and only by five written policies, with the card in hand. Out of a hundred and twenty, Kit’s search as it ships answers sixty; the full Kit sixty-one; the plain search box fifty-six; Graphiti forty-three; nothing at all, thirty. Under a plain prompt nobody was coached for, the same order holds: sixty-seven, sixty-five, fifty-nine, forty.

Two honest sentences go with that. Kit’s lead over the plain search box is real under both prompts and is not yet settled: on a hundred and twenty questions a paired test cannot rule out luck, and the lead falls short of the ten-point bar we registered before the exam was written. Its lead over the graph rival is beyond doubt. Both sentences, with the intervals and the tests, are on the page built for people who do not trust numbers, and the numbers there are rendered from the sealed artifacts rather than typed.

The trade shipping code made

While this was running, a second builder was rewriting the resolver for a different reason: it kept paging the operator. Every conflict the old resolver could not settle became a chore for a person, thirty of them on this corpus. The new one decides only where it is sure, holds the rest open as uncertainty the reader can see, retires nothing, and pages nobody.

We measured that too, because it is what ships. It gives back a few answers, mostly on the questions that need two notes from different months read together, which a merged note used to join ahead of time. The difference is inside the exam’s noise. What it removes is not: a wrong merge that has already forgotten its sources can no longer happen, and nobody gets thirty review chores per thousand notes. The next slice writes a plain reconciliation note for each open pair, both readings, both dates, composed by a template so it cannot invent, and the exam will say whether that closes the gap.

How to check it

The corpus, registration, harness, graders, cost model and statistics are in Kit's private repository. The linked evidence page displays the method and checksums, but that is not a public checkout and its commands require those files. The separate public 17-question demo exam is available to rerun; it is not this 120-question study. The questions and the answer keys are not published and will not be, because a published quiz is a quiz that can be fitted to. That costs something, and the page says what: nobody outside can reproduce the headline number. What an outsider does instead is bring their own corpus and their own quiz, seal them the same way, and publish their seal beside their numbers. That is the number Kit intends to quote outward, and we do not have it yet.

Everything above happened between Sunday evening and Tuesday afternoon, every fix measured alone against the same sealed questions, and the whole bill, including the rival’s, is on the page. The number that matters most is still the one we have not settled: seven points over a search box, on a hundred and twenty questions.