Part 6 of 7 · The loop that learns · How one developer's AI loop remembers, verifies, and enforces
okl, the tool this series is about, is meant to be a loop. Lessons reach the agent before it starts, a check proves them, and the store that holds them stays true as the code moves. That last part asks a lot. The agent has to know when to record a lesson, update it, retire it or prove it, and when a mistake that keeps coming back should turn into something louder.
So I wanted two answers. Does okl keep code quality improving over time, and does it tell an agent when to touch the store? I had agents review it against that intent, and because this series argues that no step should grade itself, I had the review checked too. The short version: the read side works, the write side only adds, and my own summary of the review needed 32 corrections.
In Claude Code, the lessons arrive on their own. Before every prompt, a hook queries the store with the prompt and puts the matching lessons in the agent’s context. Other agents still have to ask. When okl runs locally and cannot answer, the prompt is blocked. That matters because agents skip what they have to fetch, as part 5 showed. In an extreme case from a 2026 preprint by Swapnanil Saha, an agent given a seeded store and memory tools made zero memory operations in 114 turns, in a single run.
The review found two gaps. With a shared server, the hook tries a hung server three times for ten seconds each. That outlasts the 30 seconds Claude Code gives the hook, so the prompt goes through unbriefed. And when you switch okl off to work offline, the agent is never told it was not briefed. Part 3’s “fails closed” holds for a local store, not for a server that hangs. (Update, 10 October 2026: okl 0.8.2 closes the first gap. A shared server that stalls or hangs up now blocks the prompt, with the reason, inside the 30 seconds. A DNS lookup that hangs is still not bounded.)
A lesson earns its verified mark from a check. okl verify runs the command, checks the exit status and any expected output, and stores the command and the commit on the lesson. okl record refuses a mark the agent awards itself. Part 2 left one route open, for historical receipts: 160 of the 183 lessons that ship with okl arrive marked verified, with no check this store ever saw. In my own store on 10 October 2026, 70 of 295 lessons record a check that ran and passed. Only 15, all recorded in sessions, name the files they govern, so only those 15 go red when that code changes.
The measured effect holds, within its limits. I pooled the eight committed A/B runs that share one setup. Claude Sonnet wrote the code and Claude Haiku graded it, with three samples on each of eight tasks written to invite a known defect. Without the briefing, 78 of 186 attempts reproduced a known defect (42%). With it, 13 of 188 did (7%). Part 1 quoted the first of those runs, 33% against 4%, and part 5 the latest, 43% against 8%. Other cuts of the receipts give 41 to 42% against 6 to 8%.
The pool spans several versions of the retrieval code, and with three samples per task the size of the gap is imprecise. The unbriefed arm never changes, yet it read 33% to 50% across those runs, a spread the eval report treats as noise. Three of the eight tasks never produced their defect unbriefed in any of the 13 committed runs, so they cannot show a benefit. One of them, the rate limiter, produced it only when briefed, twice in 30 attempts. Storing auth tokens in the browser accounts for 16 of the 19 briefed failures across those 13 runs: 28 of 29 unbriefed, 16 of 29 briefed, reduced rather than prevented.
Corrected 2026-10-23, after an independent verification and a review. The pool is now the seven runs through the one before the layout change: 68 of 163 attempts unbriefed against 11 of 164 briefed, the same 42% against 7%; the eighth run changed how the briefing is laid out and is reported beside the pool in the eval report. Two tasks, not three, have never produced their defect unbriefed (46 to 48 attempts each); the rate limiter has produced it once unbriefed in 41 and twice briefed in 44. The unbriefed arm has since moved 21 points between two runs with nothing changed, so the floor is at least 21.
I wrote those tasks from lessons already in my store. They measure one-shot generation, not the full loop, and every run briefed from a smaller set of lessons than okl ships today. They show that a briefing cuts known defect classes on tasks built to invite them. They do not show code quality improving over time. Nothing I have measured does.
The write side rests on a single question. Once per session, at the first stop when the working tree is dirty or a commit landed in the last hour, the Stop hook asks whether anything worth recording was learned. It asks about new lessons, never about the ones the agent was shown.
My own store on 10 October 2026 shows the result. It holds 295 lessons: 110 recorded in work sessions over about eleven weeks, the 183 that ship with okl, and two shipped decisions I later dropped. Sessions linked a lesson to an earlier one only twice, both times to say “this recurred”. None of the 110 was retired, superseded or given an expiry. The only retirements were the two dropped decisions. Both were retracted, both are still briefed as AVOID, and neither is linked to what replaced it. The review found four reasons.
The briefing prints titles, not ids. Only a hint to prove an unverified lesson carries one. So an agent has to search before it can update or retire a lesson it was just shown.
Refining a lesson wipes its proof. The documented way to refine a lesson is to record it again under the same id, which replaces the whole row. The verification and the creation date are gone. Unless the agent repeats every governed file, the files go too, and the lesson is no longer flagged when its code changes.
Retirement has one blunt lever. The only status that changes how a lesson is briefed is “retracted”, which the briefing renders as AVOID: do not restate this as fact. An expiry only demotes a lesson. The store has a “supersedes” relation, but the briefing never reads it.
On 3 October I moved a docstring decision out of the Python pack that okl init imports into Python repos. It was okl’s own choice, shipped org-wide, so I recorded it as a decision for okl alone and retracted the old record. On 6 October one briefing carried both. It told the agent to AVOID “Docstrings follow Google style, and only their presence is enforced”. It also listed “okl’s docstrings follow Google style, and only their presence is enforced” as a decision made on purpose. The old one had been superseded, not proven wrong. The agent was told the same sentence was false and deliberate.
Repeat mistakes never escalate. Recurrence links are written by hand, and nothing turns a repeat into a gate. The second of those two links came out of this review. While fixing an eval script that had written false rows into my store, I added a test that opened a SQLite connection with Python’s with block. That block commits but does not close. The Python pack okl ships already held that lesson: from Python 3.13, an unclosed connection raises a warning that strict pytest turns into a failure. It was not briefed, because the prompt that started the work was the word “ok”. Its briefing held two lessons that happen to contain “ok”, and nothing briefs on the code being written. CI caught it.
Across the review’s research, three points keep coming up. I cite only sources that were checked against the sentence that uses them.
Write from outside signals. Huang and colleagues found that language models struggle to correct their own reasoning without external feedback, and sometimes do worse after trying. Kamoi and colleagues’ survey finds that self-correction works well when reliable external feedback is available. Two review bots show what that looks like in a product. Cursor’s Bugbot turns downvotes, replies and reviewers’ comments on issues it missed into candidate rules, then can promote or disable a rule as signal accumulates. CodeRabbit learns from replies to its comments, and keeps the learnings it infers itself away from other pull requests until the source one merges.
Curate rather than accumulate. In Xiong and colleagues’ study, storing every task gave a synthetic regression agent a 55.48 success rate. Strict selective addition gave 70.95, and a memory that never changed gave 67.53. Storing everything lost to strict selection for all four agents tested, though that filter uses ground truth and so is an upper bound. VibeMemBench, from part 5, points the same way for coding agents: 11 of 12 pairings of memory system and agent did no better than no memory when the systems built it from a repository’s history.
Check memory against the current code, when it matters. Copilot Memory lets its agents write their own memories, each with citations to the code. Before using one, Copilot checks the citations against the current branch and uses only facts that pass. GitHub chose that check at read time over an offline curation service. MemTrace, a research system, anchors stored history to files, symbols and tests, and checks it against the current repository before reusing it. Timing matters too. When Facebook ran Infer once a night, the fix rate was near zero. At code review, with the same false positive rate, it rose to over 70%. That is an experience report, not an experiment.
No product the review surveyed marks a memory verified only after a check its owner chose has passed, or fails a push when the code under a memory changes. That difference is narrower than it sounds. MemTrace checks memory against the code, one setup in part 5 warns when code moves past its spec, and only 15 lessons in my store can fail a push. Where okl falls behind is the write side. Bugbot and CodeRabbit learn from people’s reactions and replies. okl’s write trigger reads neither user corrections nor review feedback.
The review ran as multi-agent workflows. Four critics proposed changes, merged into 30 recommendations. Two skeptics then checked each one independently, one against the code and one for its value at its cost. A session limit cut the first run short before nine recommendations had both checks, and a second run finished them.
The skeptics confirmed every problem was real, but disagreed with the critics about the fixes. Of the 13 recommendations the critics marked “do next”, the skeptics kept one, fixing the refine that wipes a lesson’s proof. They moved the rest down, many narrowed to a cheaper first slice. One proposal would have counted every lesson with a check as armed, in part 2’s recurrence-after-arming measure. The skeptics showed it would raise a false alarm on day one: the only repeat a session had recorded came before the lesson had a check.
Then came the synthesis, which my main agent session wrote from the saved results, outside the checked workflow. It was the one step nobody had checked, so I had an independent agent check it against the data and the code. It listed 32 corrections. Eleven statements were wrong, eleven overstated, three left something out, and seven were slips such as miscounted section headers. A few were only out of date, because the finishing run’s votes came in later.
The wrong ones had a pattern. In 11 recommendations, my summary described the critics’ original proposal rather than what the skeptics endorsed. Three of the eight triggers in my table of when an agent should touch the store were mechanisms the skeptics had already rejected, and the finishing run’s skeptics rejected a fourth. The summary said the hook fails closed, when both skeptics on that item had reproduced it failing open with a shared server. Most of this needed no new evidence: the votes were already in the files the summary was written from. A summary is a step like any other, and mine had graded itself.
The review also found a false claim in my public text. The okl README, the eval report and this site listed an IDOR among the defects the A/B reproduced. That task never reproduced in any of the 13 committed runs. Even the fix needed checking: my summary’s version named four reproduced tasks when there were five. The README, the report and the project page on this site now name all five.
And agents told to be read-only still wrote. One wrote a receipt file into the repo. During the same run, eight identical briefings appeared in my store in about a second, as if a session had seen them. Both were deleted. Tracing the rows turned up a related leak: an eval pre-flight script logged every briefing it fetched. It now turns its logging off, with a test, and that test is the one that left the SQLite connection open. “Read-only” in a prompt was a preference, which is part 3’s point. The finishing run used a copy of the store, and I found nothing it wrote to the real one.
An agent should touch the store when a signal from outside the agent says to, using the exact command, at the moment it applies. It should not reflect on its own work every turn. okl link and okl verify exist today. update and retire do not exist yet; this plan adds them. (Update, 10 October 2026: okl 0.8.3 adds both, and every briefed lesson now carries its id. okl retire keeps the reason, and a replaced lesson leaves the briefing with a pointer to its replacement.)
okl link <new> RECURS_IN <old>, and the briefing says it came back with nothing checking for it. A check counts as arming only if it existed before the repeat.okl retire <id> --reason, with --by <new> so the briefing points to the replacement. A wrong lesson stays briefed as AVOID, and a fixed defect stays briefed as a warning against its return.okl update <id> merges the fields given and keeps the proof unless the governed files change. Recording over an existing id refuses without --replace.Neither verb is a new idea. Mem0 has a model add, update or delete memories as new facts arrive, and Every’s refresh command from part 5 updates, merges or deletes notes. okl’s versions have to carry a lesson’s proof through the edit.
Part 5 named two gaps as next: briefing agents other than Claude Code, and briefing again before a governed file is edited. The review ranked both below the write side. Edit-time briefing waits for a test that proves the lesson arrives before the edit, and other agents are on the review’s not-yet list.
To see whether any of this works, I need history. Today okl logs one thing on purpose: which lessons each briefing showed. Git keeps the passing checks of lessons that govern files, in the drift snapshot CI reads. Nothing records a failed check or why a lesson changed. The core I plan to add is small:
It keeps ids, counts, scores and reasons, never prompt text or code unless I opt in.
What I will not build:
Recording is the easy half. The hard half is knowing when a briefed lesson was wrong, replacing one without losing its proof, and noticing when it comes back. The review needed what okl needs: a second look from something with no stake in the first answer.
Store figures come from my own store on 10 October 2026, one person’s store whose numbers move daily. Counts about the review, such as the 30 recommendations and 32 corrections, come from my own review notes of 9 and 10 October 2026, which are not public. The A/B figures come from the committed receipts. The pooled 42% against 7% is my own sum of eight of them, which the eval report does not state: ab-20260830-0003, ab-20260901-0133, ab-20260901-1238, ab-20260902-0538, ab-20260903-0157, ab-20260903-1214, ab-20260903-1323 and ab-20260926-0528. The 13 committed runs add a cheaper generator, a different judge and three smoke tests. Each source below was checked against its text.
Update, 20 October 2026. okl now describes itself as verified memory for coding agents, and the loop that keeps it true: the lesson is the thing, its proof and its expiry are what separate it from recall, and the gate is what makes the proof matter. The review’s open items, and a prior-art read of twelve context and memory tools on 19 October, became issues #228 to #243; the first to ship should be #231, which turns an expired lesson into drift rather than a note in the briefing. The README was cut to a short version, with the comparison from Part 5 and the verification argument from Part 2 now at docs/WHERE-THIS-SITS.md and docs/VERIFICATION.md.
Papers
Products and documentation
Engineering practice and security
okl
← Part 5: What keeps an agent setup true · Part 7: The papers caught up →