Part 7 of 7 · The loop that learns · How one developer's AI loop remembers, verifies, and enforces
Last week I asked myself a question I had been avoiding: is okl anything more than a well-built instance of a crowded idea? “Brief the agent, record lessons, keep a rules file” is what every memory tool and every vendor’s org settings do now. If that is all it is, the measured numbers are nice and the project is a footnote.
The honest way to answer was to read, so I read seven papers from 2026 on memory for coding agents, from the PDFs rather than the abstracts, with the page numbers noted. The short version is that the field moved toward the premise this year, from three directions, and left one number for okl to produce about itself: how well its drift detects a stale lesson, measured the way the staleness papers measure it. The long version, with affiliations, figures and a column saying where okl stands beside each result, is now a page in the repo, docs/RELATED-WORK.md. This post is what I took from it.
A group at Notre Dame, IBM Research and Tencent AI Lab measured what I had only asserted. With a memory layer in place, 57.5% of a user’s applicable preference checks were still violated: the correction was stored, retrieved, and not followed. Their system compiles each correction into a runtime check the agent must pass before it can finish, and repeated violations fell from 100% to 37.6% on held-out in-distribution tasks and to 2.0% out of distribution, while task success held. Their one-line diagnosis is the one I would have paid for: the issue is not forgetting, it is that memory does not force compliance.
A second paper, from the University of Washington, compiles instruction files into executable checks, Tree-sitter queries for code patterns, shims that intercept forbidden commands, validators for layering. On 300 SWE-bench Lite tasks that reached 88.3% constraint compliance against 67.0% for the same rules as prose. The check is the thing. The prose is how you remember why the check exists.
That is the premise of the whole series, measured by people who had never heard of it. okl’s version has one difference worth keeping: the check lives beside the lesson, is committed, and gets re-run and gated when the code it covers moves, rather than generated at session time.
The paper I liked most came from Carnegie Mellon. They built a deterministic lab where the correct action is to do nothing, planted a guardrail for a failure class that provably never occurs, and let a self-improving harness optimiser loose on it. On input that merely looked rule-shaped, the optimiser invented a violation and added a guardrail for it in 15 of 60 runs, against 0 of 60 on featureless input. Three conditions had to co-occur: a rule-shaped pattern, an open-ended rule set, and an instruction that presupposes failures. Under an add-only acceptance loop the invented rule stays forever.
I read that and went to look at okl’s own end-of-session hook. It asks “did it surface a lesson worth keeping?” That presupposes a lesson. The store is an open-ended rule set. A session that edited a linter config supplies the rule-shaped pattern. All three conditions, by design. The hook does already accept “nothing durable” as an answer and lists the session’s failed commands as candidates, which helps, and nothing today measures whether it produces lessons on sessions where nothing was learned. That measurement is now an issue, with the reworded question waiting behind it.
The reason okl mostly survives this paper is the rule from Part 2: a stamp with no run behind it is unproven. Their fix has the same shape. They call it warrant-aware acceptance, admitting a rule only when a real violation warrants it, and it excludes the phantoms entirely.
A KAIST and NYU group pooled memories across six coding benchmarks from different domains and asked what carried over. Average scores rose 3.7%, and the gain came from meta-knowledge, validation routines and operational know-how, rather than task-specific code. High-level insights transferred. Raw trajectories were the weakest form, below no memory on two of the six benchmarks and barely above it overall; the paper’s own abstract says they often induce negative transfer through excessive specificity.
okl’s store is organised around exactly that distinction: a lesson is a rule, a decision, a gate or a defect with its cause, never a transcript, and org scope is for the ones true in any repo. The repo has its own receipt for the negative-transfer half. In September, applicability was briefly derived from the tags that record where a lesson was found; that hid 35 of 172 org records from one repo and measured worse than no filter. It was reverted the same week.
One claim I had never made follows from this. The 42% to 7% result from Part 6 is a transfer result. The eight tasks are fresh code written from a task prompt, and the defect classes they invite were learned in other projects, arriving through six of the twelve seed packs. A lesson paid for in one codebase prevented the defect in code written for another, in seven of the eight tasks; the eighth’s lesson ranked below the retrieval cutoff for its wording, which the report records, so that arm measured the briefing without it. The README now says so, with the exception.
Two well-cited papers, ReasoningBank from Google and Illinois and ACE from Stanford and SambaNova, both at ICLR 2026, curate memory automatically: the agent reflects on its successes and failures and writes the playbook itself, judged by itself, with no ground-truth labels. Both report real gains, up to 20% relative improvement in one case and about 10 points in the other. Both admit a lesson by reflection rather than by a check that ran. Read beside the Carnegie Mellon result, that is the open question of the field: does admitting lessons only when a check went red and then green keep those gains and remove the fabrication? Nobody has run it. okl has the harness, the judge-not-generator rule and the receipt discipline to run it, and that experiment is now the third issue.
ACE also found that concise summaries lose domain detail and that iterative rewriting erodes a context over time. Both argue for keeping lessons whole, which is what the briefing does, and for measuring the twelve-record limit instead of assuming it, which was already on the list.
I wrote, in chat, that nobody measures staleness. A benchmark from August does, as a diagnostic: it reports the rate at which an agent used a stale memory, 0.200 for its reference system, alongside repeated-error and over-scope rates. It also reported its own headline comparison as null, with the statistics shown, and narrowed its claims accordingly. That is the standard I want these posts held to, so the correction goes here in full.
What it measures is stale use by the agent. None of the seven papers I read measures whether a memory system detects that a stored lesson has gone stale before it briefs, which is what drift is for. (Correction, 22 October 2026: an independent verification of this post found that stale-memory detection is measured elsewhere, with precision and recall, in work I had not read. The gap I claimed is not a gap in the field; it is a number okl does not yet have for itself. The repo’s related-work page tracks the papers.)
Three things are now filed, each citing the paper that prompted it: measure fabrication under the end-of-session question, then lead with the evidence question; add stale-use and over-scope rates to okl metric; and run verified against unverified auto-curation, with the fabrication lab’s featureless pool as the control.
As for the question I started with. The idea is crowded. The specific shape is not: a lesson that carries its evidence, is re-proven in every repository that loads it, and blocks the push when its code moves, whichever agent wrote the code. The papers supplied the outside evidence for the premise. The receipts still have to be mine.