What they found

Evidence correction, 9 September 2026. The live public HTTP check on 9 September still returns neighbours on hard absence queries, although the metadata labels them absent. The newer claim below about refusal should not be read as a guarantee of an empty result from every deployed route. See the current limitations and dated public test results. The original prose and narration are retained as a historical record.

There is a button on the front page of this site that says Copy a prompt for your AI. It hands your agent the doors to two live Kits and asks it to say what it makes of them. On Thursday evening Peter pointed three of them at Kit at once, from three different labs, and then did the thing that makes the button worth having: he gave them a key to the real system rather than the demonstration.

They found a way into memories the key was not entitled to. This note is what they found, what we did about it, and how you can check that we did.

The one that mattered

Kit lets you share one knowledge area with someone else’s Kit. The key carries the limit, and the limit is supposed to be enforced when the memory is read rather than when it is written. That is the whole mechanism, and the public demo has an exhibit built to show it working.

It worked when a memory was searched for. It did not work when a memory was opened by its id, or asked for through the routes one Kit uses to answer another, or reached through a second Kit’s results. Four ways in. One of them was guarded. An agent holding a key scoped to a single project read medical history, a home address, a purchase price and another client’s project structure, and was right to call it a release blocker.

Nothing private left anyone’s machine that should not have: the reviewer was reading a Kit whose owner had handed over the key deliberately, and it reported what it saw rather than repeating it. But the authorisation path is the one Kit asks people to trust with real memories, so the severity is not measured by what happened to leak this time.

All four are closed. The fix is not the interesting part; the reason all four existed is. Each was the same question asked at a different door and answered differently, and nothing in the test suite had ever asked a Kit to refuse anything. Every test asked whether Kit works.

So the release now has a stage that mints a key limited to one project, writes four memories into a brand-new Kit, and asks for each of them three ways. Two must come back and two must not, because a fence that refuses everything would pass a test that only checks refusal, and would quietly cost every reader their own context. That check runs on a fresh install before any build reaches you, and the release stops if a scoped key gets through.

The cause underneath it

While repairing that, we found the reason personal memories were reachable at all. Kit summarises your sessions, and it correctly recognised which of them were personal and tagged them so. Then it filed each memory using the scope of the conversation it came from, and on a working install almost every conversation counts as work.

So everything Kit knew to be personal was stored as work anyway. On the first Kit we checked, that was 3,764 memories, and not one of them was filed the way its own tags said it should be. A conversation holds a deployment decision and a doctor’s appointment; Kit could tell them apart and then threw the answer away.

That is fixed at the source, and fixing it alone would have made things worse. The new read fence refuses a personal-tagged memory to anything that cannot see personal scope, so a memory the old code filed as work would have been refused to the work view by its tag and to the personal view by its column: invisible to both, and only reachable by an unrestricted key. Your Kit repairs its own on the first start after updating, and leaves alone the ones tagged both personal and work, because your working preferences are genuinely both.

The criticism all three made

Independently, from three labs, in almost the same words: Kit does not know when it does not know. Ask it about something it has no record of and it returns the nearest few memories, which read like an answer because they are confident, relevant-looking and wrong. Our own published evaluation scored zero out of four on exactly that.

We built a way to measure it before trying to fix it, and the first version of the measurement flattered us by sixty points. It invented names sharing no vocabulary with the archive, which any retriever rejects. The honest version asks about a real subject and a fact the archive does not hold, which is the only shape an honest question takes by accident. On that set Kit scored zero.

Measuring also killed the fix we had planned. Gating on a confidence score does not work, because a question about a real subject matches its subject strongly: fifteen of twenty unanswerable questions scored above the weakest genuine one. Averaging a matched subject with an absent fact tells you nothing about either.

What separates them is not a number but a list. Ask when someone travelled to Lisbon and the word Lisbon appears nowhere in the archive. Ask which regiment they served with and neither does regiment. Ask a real question and that list is empty. So Kit now asks its own store two questions directly: has it ever held this word, and does it hold anything connecting this word to the subject you asked about. Neither is a threshold. Neither is a judgement. Neither is another model asked to be careful on our behalf, which would move the burden of honesty without discharging it.

Honest refusals on the hard set went from zero to one hundred per cent on the closed archives and seventy-nine per cent on a five-month working Kit of 62,000 memories, with no loss in answering the questions it can. The remaining twenty-one per cent is incidental co-occurrence in a large corpus, and we would rather publish that number than round it.

The same check does a second job we did not expect. When the archive does connect the terms and the search simply ranked the answer badly, Kit now says so and names the memories it should have shown you. That is a retrieval failure being reported rather than hidden, which is the opposite of what a confident wrong answer does.

What we did not fix

The reviews found more than this, and the rest is real. Consolidated memories record which inputs were in the batch rather than which ones the conclusion actually rests on. Session extractors write the same fact three or four times and the duplicates crowd out better answers. Memories that have decayed still occupy the top results. Most memories carry no path back to the transcript they came from, even when the commit is written in their text. A memory recalled from another Kit cannot be reopened by its id.

None of that is fixed today. It is written down on one page, along with everything else we know is wrong, and that page existed before these reviews and was found and cited by two of the three.

The receipts

You do not have to take any of this from us. The archives are live and keyless. Ask one of them about something it does not hold and read what comes back: it should name the words it has never recorded and say there is no answer, rather than offering you a neighbour. Ask about something it does hold and it should answer normally. Both are one fetch, no account, nothing installed.

The evaluation prompt is still on the front page, and the sharper internal version of it is the one that found all this. If you want to run it again, we would rather you did. The version of Thursday that would have gone badly is the one where we shared Kit with friends first and a scoped key went out with the door open.

Everything above shipped between Thursday evening and Friday morning, in five releases. Every claim in this note has a number attached because a project asking you to trust it with your working life should be checkable, and the number that matters most is the one we have not fixed yet: twenty-one per cent.