Known limits
Kit is an early beta, and a memory you cannot check is worth nothing, so this is the list we would want if we were deciding whether to try it. Everything here is known, reproducible, and ours to fix. If you find something that is not on this list, we would like to hear it.
Reviewed 23 October 2026. Each entry says when its figures were last checked.
On LongMemEval-S, a public 500-question benchmark, Kit scored 84.4% and a plain search 83.0%, one run each: a tie, not a lead. Our own 120-question study uses sealed questions in a private repository, so its headline cannot be checked from its page, and the public 17-question exam still has failures (read the dated results).
Ask about something that is not in memory and recall can still return the closest records it has, while its own answerability signal says the answer is absent. Read the signal, not only the hits, and do not turn a similar record into an answer.
When two federated Kits assert different things, both come back and both are marked contested. That part is deliberate, because a memory that silently picks a winner is worse than one that shows you the argument. It is also unfinished: there is no trust-per-claim policy yet, so deciding which source to believe is still your job.
They are seeded archives with the questions, rubric and explanatory answers written by us in advance. Explanations are released after live evidence checks, not generated by a model. They show that memory can be stored, scoped, corrected and retrieved live. They do not demonstrate ingestion, nightly consolidation, decay, handoffs, backup or restore, because those happen inside a running Kit over weeks. Use a bounded trial to check those behaviours in the app. No published customer study establishes the benefit for your workflow.
Claude Code, Codex, Cursor, VS Code and the other coding agents are captured live as you work. Files, email, notes and your Mac's calendar come in through the app. Browser chats are not captured live yet: a ChatGPT or Claude export brings them in once. The supported page marks which is which.
Since 0.2.440 Kit ships compiled, so its code is not open to read. It is not open source, and nobody outside this project has audited it. There is no SBOM and no published threat model yet. The download is signed, notarised and checksummed, which is a different and much smaller claim.
Every night Kit's full test suite runs against the release. On 23 October, against 0.2.446, it passed 17,091 tests and failed 2. No night has been fully green since 4 September, and releases ship while it is not. Until it is, a release note says what changed, not that nothing else broke.
The release notes on the install page are complete rather than curated, so they include the month Telegram was broken on every personal install, the reflection cycle that had not run since the runtime migration, imported text that could execute code inside the app, a local database that accepted connections without a password, files that other accounts on the same Mac could read, and builds before 0.2.271 in which a read-only key could call routes that change data. All fixed, all recent, and each fix reaches an install when it updates; the update that closed the last of them cannot be skipped. The rate of serious fixes is the honest signal about maturity, and it is why the advice below is what it is.
Kit is designed by Peter Koen, a user experience designer, not a career programmer. Since March 2026 he has built it with AI coding agents: he decides what Kit should do and how it should feel, the agents write and fix the code, and he checks the result against tests and real use. Testers have used it every day for weeks, the longest for months. That is why the test suite, the release notes and this page are as long as they are: they are how a designer makes sure code he did not type does what it says. It also means someone with a security or systems background will find things we have not. When you do, please tell us.
What we would tell a friend: try it on one real but non-critical project, keep the original sources, and check what it remembered rather than assuming. Do not make it the only copy of anything yet.
Kept here because a list of what is wrong is only believable beside a list of what was wrong.
Recall across linked Kits always named the Kit each result came from, but following that citation back opened your own record with the same number, or nothing. Every Kit now cites the same way, Name:m#N, and asking for Shiro:m#7 opens Shiro's memory 7.
A key limited to one knowledge area was correctly limited on its own Kit, and peer results were merged without that limit applied, so a federated query could return areas the key had never been granted. This is the mechanism Kit offers for sharing one area with a colleague’s Kit, so it mattered a great deal more than the fictional data it exposed.
Found by an unaffiliated AI agent evaluating the public demo. Reported, reproduced, fixed and verified the same day. Peer results now face the same scope check the local ones do, on our side of the connection, and tests pin every merge site so it cannot come back quietly.