Does a lesson change what a coding agent writes?

A methods paper · 12 November 2026 · the primary record is okl's evaluation report

Abstract

On five real defects, a coding agent given no lesson fixed none, and one given the right lesson fixed about two in three. Given the same lesson in a list of ten, it fixed about one in seven.

I study okl, which keeps a coding agent true to a team’s rules, specs and docs, and keeps those true to the code. okl calls each thing it holds a lesson: a rule, decision, convention or doc claim, some written as policy and some after a defect. This study tests the first half of that claim. When a lesson applies to a change, does putting it in front of the agent change the code it writes, and what limits that?

The main method is an outcome replay. I take real defects from the project’s history and restart each one from the change as first written, before review caught the flaw. Defects are the test bed, not the point: they are where a lesson’s effect can be graded, because the real fix supplies a test. An agent finishes the change under a controlled condition. A hidden test, derived from the real fix, grades the result by its behaviour. Three of the five graders have a positive control, so a fix that refuses everything fails them.

In pre-registered pilots (Sonnet as the agent; 25 sessions or picks per arm, 75 to 150 per experiment):

Separately, two judge models agreed on whether a change broke a lesson with Cohen’s kappa 0.742. Appending yes/no conditions to each lesson did not raise agreement (0.599; change −0.143 [−0.368, 0.094]).

The tasks are few, from one repository, and chosen because a lesson applies. These are pilots that decide what to test next, not effect sizes to generalise. I describe each method in enough detail to rerun it on another project’s history, with the integrity rules that kept the results honest.

Plain-language summary

A written lesson works on an AI coding agent much as a protocol note works on a new lab member. Read alone and at the right moment, it changes what they do. Buried in a binder of ten, it is mostly skimmed.

The setting. Coding agents are language models that edit a codebase: they read files, write code and run the tests. They start each session knowing nothing of a team’s rules, specs, architecture decisions or docs, and nothing keeps those in step with the code. okl does both jobs. It holds what the team maintains as lessons: a rule, a decision, a convention, a doc’s claim, or a warning (“a gate that checks files must read the committed files, not the working copy”). It puts the ones that apply in front of the agent, and it flags when the code moves away from what one describes. Recording what each session learns comes third. A briefing goes in before the work starts, chosen by matching the task’s words. A review runs after the code is written, with the agent checking its change against lessons one at a time.

The experiment. I test the first job: does a lesson in front of the agent change its code? Past defects make good cases because the right answer is known: a lesson covered the defect, and its real fix added a test. Each is treated like a case in a controlled trial:

  1. Take a real mistake from the project’s history, at the moment just before a reviewer caught it.
  2. Hand that half-finished change to the agent under one condition (no lesson, the right lesson, the lesson in a list of ten, and so on).
  3. Grade the result with a hidden test built from the real fix. The test checks behaviour and, where it can, includes a control case so that a fix which refuses everything fails.
  4. Repeat each condition five times, in a random but fixed order, and compare pass rates.

The plan, the conditions and the pass bar are written down and committed before any session runs. That makes the study pre-registered: any later change to the plan is a dated amendment, written before the data it affects is seen.

What I found. The lesson itself works: with it, the agent fixed most defects; without it, none. How the lesson is handed over matters as much as its content: a list of ten cut the effect by two thirds or more, and asking about each lesson separately restored it. The hardest remaining step is choosing which lessons to show. The best methods I tried find the right one about half the time, and most find less.

How confident to be. Moderately, about direction; not about size. There are five tasks from one project, one agent model, and five repeats per condition. The results decide what to test next. They are not a general law about AI agents.

Background and the questions

Whether written guidance changes a coding agent’s work is an open empirical question, and recent studies disagree on the answer.

On repository instruction files, Gloaguen et al. found that context files do not generally raise task success and add over 20% to cost. Xu and Wu found the opposite failure: agents followed a stale file over the code in 90.1% of choices. One added sentence moved code-following from 4.2% to 91.4%. SkillsBench found curated skills raised average pass rates from 33.9% to 50.5%. Skills the agent wrote for itself scored below none. Tsuchida et al. found that conditional wording (“if X, do Y”) was followed less than prescriptive or explanatory wording.

Reach is not compliance. okl’s first evidence was a behaviour result on authored tasks: on eight tasks written to invite defects the store already knew about, the briefing cut defect reproduction from 42% to 7% (68 of 163 against 11 of 164 attempts, seven runs, graded by a model judge). Later studies on real pull requests measured reach: whether the right lesson appeared in the briefing. On 58 of them, lessons the briefing showed were followed no more often than lessons it did not: 63.3% against 72.3%, a difference of −9.0 points [−22.0, 4.2]. That observational result is a correlation and could hide an effect. It still meant that reach alone could not be the measure.

Model-made answer keys were not enough. The real-work studies graded “was this lesson followed?” with a judge model. A third model, asked to settle 24 disputed cases, did not: it called only 2 of the 4 agreed-broken controls broken. Agreement between models predicts whether a labelling task suits them at all (Ahmed et al.), and judges favour their own output (Pombal et al.). So I needed outcomes that do not rest on a model’s opinion: a test that passes or fails. A later blind audit bore this out: a second model agreed with only a third of the pairs the key called broken, and no person has yet labelled them, so the results scored against that key (§4bc to §4be) are scored against a model’s key (REPORT §4bh).

The three questions. Each experiment below answers one:

  1. Does a lesson change the code at all, and does its wording matter?
  2. How should lessons be presented: alone, in a list, or one at a time?
  3. Which lessons, and when: can okl choose the right one from the task, a plan or the finished change?

The first two are about compliance and the third is about retrieval. The results show that they fail in different ways and need different fixes.

The system under test

okl does three jobs, in this order: it keeps a coding agent true to a team’s rules, specs and docs; it keeps those true to the code; and it keeps learning as the team builds. It is local and git-native, and runs no generative model by default. These experiments test the first job. Most hand the lesson text to the agent directly, to isolate the effect of the lesson itself; one (§4az) runs okl’s own review after writing, and one (§4as) observes the briefing on real work. The read-time briefing was not tested. All of them use one repository, okl’s own, with about 300 lessons.

TermWhat it isWhere it acts
LessonOne record of something the team maintains: a rule, decision, convention, doc claim or warning. It has a title, the rule, the fix, why, tags, and optionally the files it governs and the check that proves it. Some are named for the incident behind them rather than the code they govern (§4bb).The store, .okl/okl.db (SQLite), plus shared packs
BriefingLessons chosen by matching the task’s words. Full-text search fused with an embedding search; up to 5 shown in full and 12 more in one line each, plus up to 10 standing lessons.Before the agent starts (a prompt hook)
Read-time briefingLessons whose governed files match a file the agent opens.When the agent reads a file
Review after writingFive steps in okl check --change: (1) say what the changed code does, one mechanism a sentence; (2) pick the ten lessons the change most likely breaks, from every lesson’s one line, those governing a touched file first; (3) check each in its own fresh sub-agent with one question: does this change break this lesson? (4) without sub-agents, one lesson per pass; (5) report.After the code is written; opt-in
Drift gateA lesson with governed files and a recorded check goes red when those files change after the check last ran.In CI

Two definitions recur. A lesson is broken by a change when the change does what the lesson warns against, or fails to do what it requires. A lesson is found when it appears in the list a method returns, at the stated size.

Terms for readers outside the field.

Nothing in the methods depends on okl’s storage. Any system that holds written lessons and decides which to show an agent can be tested the same way.

How the work iterates

The technique is a loop: measure one thing against a bar set in advance, change one thing because of the result, and measure again against a control in the same sitting.

Each result set the next change, and each change was measured again Experiments in order, REPORT §4as to §4bc (all pre-registered but §4as), 10 to 12 November 2026. What I measured What I changed because of it §4as · Real work, observed Lessons shown were followed no more often than lessons not shown, on 58 real pull requests Build a controlled test Replay real defects, graded by hidden tests §4au, §4av · Can models be the key? Judges split on disputed cases; yes/no conditions made them stricter, not more consistent Grade by behaviour, not opinion Graders from the real fix, positive controls §4aw · Does a lesson change the code? No lesson: 0 of 25 fixed. Right lesson alone: 0.64. Rewording it as a contract did not help Keep the format; test a list A real briefing shows many lessons at once §4ax · In a list of ten? Falls to 0.12 to 0.2: in most sessions the agent edited nothing and called the change ready Hand lessons over one at a time Each in a fresh session, once the code exists §4ay · One at a time? 0.72, level with the lesson alone; about one edit for the nine others; about seven times the cost Rebuild the review (#427) Pick ten, then a fresh sub-agent per lesson §4az · The rebuilt review, end to end 0.2, against 0 for the old review and for none. Here the agent chose its own ten lessons Keep it opt-in; study choosing Choosing explains most of 0.72 to 0.2 §4ba · Choose from a plan? From a plan or the diff 0.48; from the task 0.36. Two of the five tasks never found Describe lessons by mechanism Some are named for incidents, not code §4bb · Applies-when lines? Up 0.1 on average, under the 0.15 bar. #196 and #309 still never found Choose from the code itself By hunk, description, code name and a screen §4bc · Choosing from the code Pick plus code lists, screened: +0.11 [0.04, 0.19], short of 0.15, at 22 lessons; screen kept most Make the screen tell lessons apart Ask the review’s own question; cut to ten next
the evaluation loop · the main line of experiments, §4as to §4bc (§4at not shown)

Read it in zigzag. Each box on the left is a pre-registered experiment and its result. The box beside it is the change that result caused, and the arrow down is the next experiment, which tests that change. Two of the changes were to the measurement itself (the controlled replay and behaviour graders). The rest were to okl, or to the question asked next.

Three rules keep the loop honest. A change is never judged on the data that suggested it. A failed bar is reported as failed, and the change it would have justified is not built. Each new experiment re-runs the earlier arm it is compared with, so a result is judged against a control from the same sitting, not a number from another day.

Method 1: the outcome replay

The outcome replay measures compliance by behaviour: a real defect is restarted just before review caught it, an agent finishes the change under one condition, and a hidden test from the real fix grades what it leaves.

1. Choose tasks from real history. A candidate is a past change whose real fix added a test that fails on the change as first written and passes after the fix. Read each test against the lessons that existed when the change was made. Keep a task only when the test checks what a lesson already said. Exclude for cause:

In okl’s history five tasks survived: #115, #168, #196, #198 and #309.

2. Freeze the start. Each session starts from the change as first written, before review found the defect. It runs in a copy with one commit of history and no agent configuration files. This tests an agent finishing a written change. Starting from nothing would grade the private names the real author happened to choose.

3. Apply one condition per arm. Every arm receives the same instruction: get this change ready to merge. Arms differ only in what lesson text the agent sees and how. The conditions used so far:

ExperimentArms
Lesson vs none, and its wordingno lesson; the lesson as stored; as a contract (when, do, done when, does not apply, why); contract plus examples
Listsalone or among nine others; told it applies or not (2 × 2)
One at a timealone; list of ten; the same ten, each in its own fresh session
End to endno review; old review (one pass over ten); new review (a sub-agent per lesson), with the agent choosing

The nine other lessons in a list are those okl’s search ranks highest for the task’s words, excluding the right one and any lesson that cites the task. A separate agent wrote the contracts and examples from the lessons alone, never shown the tasks.

4. Run the agent. Sonnet, one session per run, with tools to read, search and edit files and to run only pytest, git diff and git status. Five repeats per arm and task, in a seeded random order, so a run that stops early favours no arm. In the one-at-a-time arms a run is one fresh session per lesson, so each of their runs is about ten sessions, not one. Each session saves its patch.

5. Grade with a hidden behaviour test. After the session, the grader is laid over the tree it left. The agent never sees it. A session passes when every case in the grader passes. Each grader must fail on the frozen code and pass on the real fix, and that check is itself saved as a receipt.

Write behaviour graders, not the fix’s own test. Four of the five real tests called a function the fix invented. On the frozen code they failed on a missing name, so no agent could pass without guessing it. The replacement graders drive the code through the interface it had when frozen and assert what the lesson asks. For example, two commands that resolve to the same model must not start a run. The graders for #168, #198 and #309 each have a positive control, so a fix that refuses everything fails. #196’s accepts a refusal, which also keeps its lesson, and #115 keeps the real fix’s own test, which fails by assertion.

6. Grade from saved patches. Because each session saves its patch, graders can be fixed and re-run without re-running agents. The regrader reports any patch that did not apply, was cut off, or left a file outside the patch.

Measures. The pass rate for an arm is averaged over tasks, so each task weighs the same whatever its repeats:

pass(a) = 1/|T| · Σ_{t ∈ T}  #{passing sessions of arm a on task t}
                             / #{sessions of arm a on task t}

Also reported: sessions that edited anything, harm (another test in the grader’s file that passed on the frozen code and now fails), turns, time and API-priced cost. In end-to-end runs, sub-agent use is counted from the session’s own event stream.

Method 2: measuring choosing

Choosing is measured without running an agent: a method returns a ranked list of lessons for a change, and I count how often the lessons that change needed are in it. This is cheap, so it can run on many more changes than the replay.

Two benchmarks.

Lessons as they stood. Every method searches only the lessons that existed when the change was frozen, so a lesson written later cannot leak the answer. Their text, though, comes from a later snapshot, and some were extended from these same changes, which can help every method alike (see Threats).

Recall at k. For change c with broken lessons B(c), and a method’s top-k list L_k(c):

recall@k = 1/|C| · Σ_{c ∈ C}  |L_k(c) ∩ B(c)| / |B(c)|

k = 10 is the review’s own size: the review then checks each chosen lesson one at a time.

Starting points and methods tested.

MethodReadsChooses by
Taskthe task’s title and issueokl’s search (full text fused with an embedding); or a model picking ten from every lesson’s one line
Plana plan a model writes in the repository before any code, one mechanism per sentencea model picking ten
Diffthe finished change, cut to 40,000 charactersa model picking ten (today’s review)
Explanationa model’s account of what the changed code doesokl’s search
Applies-when lineseach lesson’s line plus a model-written “applies when” by mechanisma model picking ten
Chunkeach changed hunk (the 20 largest, 1,500 characters each) as its own queryokl’s search, fused
Descriptionone model-written sentence per hunk saying what it doesokl’s search, fused
Anchorcode names, flags and paths the change uses; files lessons governexact matching, rare names weighted up; no model
Screenthe model’s pick fused by rank with the chunk, description and anchor lists, top 30a small model (Haiku), asked of each lesson alone: does this change do something this lesson is about?

Fusion. Several ranked lists are merged by reciprocal-rank fusion, with the usual constant 60:

score(ℓ) = Σ_L  1 / (60 + rank_L(ℓ))

Uncertainty. Differences between methods are paired by change and bootstrapped over changes, 2,000 draws with a fixed seed. The 95% interval is reported.

Method 3: the integrity protocol

Every rule below exists because an evaluation of model behaviour has more free choices than a reader can see. Each one closes a specific way to fool myself, and several were added after one did.

PracticeWhat it preventsHow it is held here
Pre-registrationChoosing the measure or the bar after seeing the dataArms, measures, decision rule, threats and budget are committed to the report before any counted call
Dated amendmentsSilent changes mid-studyA change after registration is written, dated and committed before the data it affects is seen, saying what was already seen
A rule with a reading per outcomeReading a null as a near-missEach section states in advance what each outcome means (“collapses in a list”, “stays opt-in”) and what work it points to
Same-sitting controlsDrift in models and tools between daysEvery bar is stated against a control arm re-run in the same sitting, never against an old number
Development and test halvesTuning on the data that decidesPrompts are revised on the development half only, at most three times. The test half runs once.
Freeze by hashEditing inputs after seeing outputsConditions and lines are hashed in the script; the test half refuses to run under an unfrozen hash
Judge is not the generatorSelf-preference in model judgesThe harness refuses a run whose judge model is the generator; agreement is measured between two models that wrote neither the lessons nor the key
Blind authorshipLeaking the answer through the treatmentContracts, examples and applies-when lines are written by a separate agent shown only the lesson text, never a task
Graders with positive controlsA grader that a do-nothing or refuse-everything fix passesEach grader fails on the frozen code and passes on the real fix, a check saved as a receipt; three of the five also have a case that refusal fails
Seeded call orderStopping early favouring one armAll arms interleave in one seeded random order
Completeness thresholdsSelective stoppingA reading needs about 90% of sessions complete, a count fixed in advance; a run cut by a usage limit resumes at the same commit and prompt, and partial receipts are deleted
Run guardResults that no commit can reproduceEach receipt carries the commit, whether the tree was dirty and the script’s SHA-256
Receipts and figure testsTranscription errors and stale numbersEvery figure the report quotes has a committed receipt, and a test fails if the report’s text and the receipt disagree
Labelled interpretationPost-hoc stories passing as findingsReadings not fixed in advance are headed “What it means, not pre-registered”

The protocol has caught me several times, each correction recorded in the report beside the claim it changed. A grader filter read an empty field and excluded nothing (§4aw). A briefing size was quoted as 12 places when it is up to 17 (§4ax). An agreement figure came from cases chosen because they were disputed (§4au, corrected from §4av). A script would have read plans from the working tree, not the committed receipt (§4bb). A thread fault stopped two runs before any row was kept (§4bc).

Results

The lesson itself changes the code, how it is handed over decides whether it does, and choosing the right lesson is the largest gap found.

Presentation (§4aw to §4az). Each dot is one arm of 25 runs (a run is one session, or one per lesson in the one-at-a-time rows); a row’s spread mixes arms (told it applies or not) and sittings.

A lesson fixed most defects alone or one at a time, and few in a list of ten Pass rate on five real defects, averaged over tasks; each dot is one arm of 25 runs. 0 0.25 0.5 0.75 1 No lesson 0 Right lesson, alone 0.6 to 0.72 Right lesson in a list of ten 0.08 to 0.2 Same ten, one at a time 0.72 Review, agent picks ten, one pass 0 Review, agent picks ten, one at a time 0.2 In the bottom two rows the agent chose its own ten; choosing explains most of the drop, perhaps all.
REPORT §4aw, §4ax, §4ay, §4az receipts · 5 tasks, 12 arms in 4 sittings, 25 runs each

With no lesson, the agent left all 25 changes as it found them and called them ready. In a list it edited in 16 of 50 sessions, against 39 of 50 alone. Telling it the lesson “bears on this change” made no difference, alone or in a list. One at a time cost $0.81 a session against $0.12 alone. Its median session made one edit for the nine other lessons (whether those edits helped was not judged), and no session in any arm broke another test. Rewording the lesson changed little: as a contract it scored 0.52 and with examples 0.6, against 0.64 as stored.

Choosing (§4ba, §4bb). A picker read every lesson’s one line and named ten, after reading the task, a plan written before any code, or the finished diff.

No way of choosing ever found the lesson on two of the five tasks Share of five Sonnet picks with the task’s lesson among their ten. One line per lesson (§4ba) Task Plan Diff With an applies-when line (§4bb) Task Plan Diff #115 stored data reaching a shell 0 1 1 0.2 0.8 1 #168 judge model is the generator 0.8 0.4 0.4 0.8 0.8 0.6 #196 untracked file in a measurement 0 0 0 0 0 0 #198 run on another judge’s series 1 1 1 1 1 1 #309 database URL swapped silently 0 0 0 0 0 0 Average over tasks 0.36 0.48 0.48 0.4 0.52 0.52
REPORT §4ba and §4bb receipts · 5 tasks × 5 picks per cell

The left half and right half are separate sittings. In §4bb’s own sitting the plain list scored 0.36, 0.44 and 0.4, so the lines added 0.1 on average from a plan or diff, under the 0.15 bar. #196’s lesson is written about audits of files, and the faulty code was an eval script loading a folder. On #309 no plan named the code that resolves the store, and in the diff it lay past the 40,000-character cut.

On the 34 development-half changes with a broken lesson, okl’s search found 14.3% of broken lessons in its top five from the task’s words. It found 12.9% from the added code and 14.3% from a model’s explanation of it. A model reading that explanation and picking ten from every lesson’s line found 24.3% at five and 40.0% at ten (§4at; development half only, in-sample and not pre-registered).

Judge agreement (§4av). On 70 cases, Sonnet and Fable 5.1, each reading one lesson against one change, agreed on broken-or-not in 62 cases (kappa 0.742). With yes/no conditions added they agreed on 56 (kappa 0.599), a change of −0.143 [−0.368, 0.094].

Choosing from the code (§4bc). Pre-set reading: choosing does not improve enough. Each list was scored by the share of a change’s broken lessons it held, on the 58 keyed past changes, as a model’s key labelled them (see the audit above). The pool fuses the model’s pick with the three lists drawn from the code, and a small model (Haiku) screened each pooled lesson.

The screened set beat the pick by 0.11, short of the 0.15 bar Share of each change’s broken lessons a list held, averaged over 58 past changes (124 lessons); dashed line at pick + 0.15. Share held List size the bar: pick + 0.15 Model picks ten (today) 0.465 10 Each hunk as a query 0.261 10 A sentence per hunk as a query 0.366 10 Code names and governed files 0.090 10 The four fused 0.586 30 Fused, then screened one by one 0.577 22.1 avg The screen said yes to most lessons it was asked about, so it barely cut the pool of 30.
REPORT §4bc receipt pick-fix-20261112-0425.json · 58 changes, 124 broken lessons

The gain’s 95% interval was [0.04, 0.191]. The pool holds the pick’s own list, so part of that gain is by construction, and it came at 22.1 lessons on average against the 12 the pre-registration allowed. The screen said yes to 1,281 of 1,727 lessons. Each list built without the pick held less than the pick did. Every method still missed all of a change’s broken lessons on some changes: the pick on 19 of 58, the pool and the screen on 13.

Every experiment, with its receipt (in evals/results/):

SectionQuestionDesignReading (pre-set unless noted)Receipt
4asAre lessons the briefing showed followed more often?Observational, 58 PRs, 317 pairsNo association (exploratory, not pre-registered)briefed-follow-20261110-0426.json
4atDoes explaining the code find broken lessons?34 development changesNo better than the task’s words (in-sample; the test half was not run)explain-search-dev-20261110-1306.json
4auShould the review be on by default?20 unseen changesRule fails; off by defaultdefault-read-20261110-1438.json
4avDo yes/no conditions raise judge agreement?70 cases, 280 verdictsRule fails; not shippedconditions-agree-test-20261111-0038.json
4awDoes a lesson change the code, and does wording matter?Replay, 4 arms, 100 sessionsNo lead for the contract; the lesson’s own effect was reported with no ruleoutcome-replay-20261111-0129.json
4axDoes it survive a list of ten?Replay, 2 × 2, 100 sessionsCollapses in a listreplay-list-20261111-0249.json
4ayDoes one at a time recover it?Replay, 3 arms, 75 sessionsWorksreplay-one-20261111-1355.json
4azDoes the rebuilt review work end to end?Replay, 3 arms, 75 sessionsStays opt-inreview-e2e-20261111-1543.json
4baDoes a plan find the lesson?75 picksNot well enoughplan-reach-20261111-1435.json
4bbDo applies-when lines help a picker?150 picksNot enoughwhen-lines-20261112-0058.json
4bcCan okl choose from the code?58 changesDoes not improve enoughpick-fix-20261112-0425.json

What failed and what it taught

Most ideas failed their pre-set bar, and the failures shaped the method as much as the successes did. Each row was decided by a rule written before the data.

AttemptExpectedWhat happenedWhat it taught
Review on by default (§4au)It finds at least 30% of broken lessons17.4% [4.0, 37.5]. It picked half the broken lessons, then judged most of them followedFinding a lesson and checking it are separate steps; the check reading ten at once was the weak one
A third model to settle disputes (§4au)It agrees with the control cases both sides called broken2 of 4 controlsDisputed cases are the wrong sample for agreement; a key needs an outcome, not another opinion
Yes/no conditions on lessons (§4av, after CheckEval)Agreement between judges rises by 0.15Kappa 0.742 without, 0.599 with: a change of −0.143 [−0.368, 0.094]. Broken calls rose from 46 to 74; false alarms from 1 to 4The premise was wrong: two judges reading one lesson at a time already agreed. Conditions made them stricter, not more consistent
Lessons as contracts (§4aw)Contracts beat the stored text by 0.20.52 against 0.64Wording was not the bottleneck. A “does not apply” clause may give the agent a reason to stop
The real fix’s test as grader (§4aw)It grades fairly4 of 5 called a function the fix invented, so they failed on a missing name. The filter meant to catch that read an empty fieldRead the actual exception on the frozen code. Write behaviour graders with positive controls
A plan to choose lessons early (§4ba)At least 0.6, and 0.25 above the task’s words0.48, and 0.12 aboveA plan chooses as well as the finished change, so choosing can move earlier, but neither is good enough
“Applies when” lines by mechanism (§4bb)Hit rate up by 0.15Up by 0.1, nearly all on one taskA line written from the lesson alone keeps the lesson’s own framing, and cannot know which other code shares its mechanism
The size of a default briefing (§4ax threats)Quoted as 12 places, with the right lesson’s search ranksA briefing is up to 17 places (5 in full, 12 in one line), and the ranks came before the briefing’s own filters. Run as shipped, it showed the right lesson on 2 of 5 tasks. Corrected by amendment before scoringMeasure what the product shows, not an intermediate list
§4bc’s first two runsRows keptNone. The embedding’s tokenizer cannot be shared across threadsShared model objects may not be thread-safe; count empty results and stop on a run of them

§4bc joins this list. Fusing today’s pick with lists drawn from the code, then screening each lesson, beat the pick alone, but by less than its bar and at twice the size, because the screen asked whether a change bears on a lesson and said yes to most. A screen has to tell lessons apart, not just find related ones.

The successes were fewer. The lesson itself changes the code (§4aw). One at a time recovers what a list loses (§4ay). The review rebuilt on that beats the old one end to end (§4az).

What this means for your rules file

A rule changes an agent’s code when it reaches the agent on its own, at the moment it applies, and still matches the code. Length and wording mattered less than that in every test here. These come from five tasks in one repository, so treat them as hypotheses to test on your own history with the replay.

  1. Keep the always-on file short, and do not rely on it. A rule in a list of ten was acted on a fraction as often as the same rule alone (§4ax). Every rule you add to a file read at session start dilutes the others.
  2. Bring rules in per task, and check the change against each one separately. The same ten rules asked about one at a time did as well as the right rule alone (§4ay). A sub-agent per rule after the code is written is one way to do it.
  3. Expect choosing to be the weak step. No method here put the right rule in front of the agent much more than half the time (§4ba to §4bc). Rules are usually written about the incident that taught them, while changes are described by feature.
  4. Name the code a rule is about. Matching exact code names and governed files found few of the broken rules (§4bc), and why was not measured. A rule that names the files and functions it governs can be found from the code and checked when that code changes.
  5. Spend less effort on wording. Rewriting rules as contracts, with or without examples, did not beat the plain rule (§4aw). Conditional phrasing may be followed less (Tsuchida et al.).
  6. Give each rule a check where you can. A test, a lint rule or a script turns advice into a gate the agent cannot skim past.
  7. Keep the file true to the code. Agents followed a stale instruction file over the code in 90.1% of choices (Xu and Wu). Tie each rule to what it describes, and re-check it when that changes.
  8. Plan first if you like, but do not expect the plan to fix choosing. A plan written before any code chose rules as well as the finished change, and no better (§4ba).

One common habit was not tested here: adding a line to the file after every mistake. By points 1 and 7, a file grown that way gets longer and goes stale, so pair it with pruning and checks.

Threats to validity

The largest threat is size: five tasks from one repository, chosen because a lesson applies, run with one agent model.

Internal validity.

Measurement validity.

External validity.

What these results do not show. They do not show that lessons improve code quality in general, help with mistakes the store has never seen, or compound over months. They do not show effects on human developers, or on other languages, repositories or agents.

How to reuse the methods

The replay needs three things: a repository whose fixes added tests, written guidance with dates, and the ability to rebuild that guidance as it stood at any commit. A first pilot of 100 agent sessions cost about $12 at API prices, a median of 77 seconds a session.

Outcome replay, step by step.

  1. Mine candidates. List past changes whose fix added a test. Keep those where the test fails on the change as first written and passes on the fix. Print the actual exception on the frozen code: a missing name or import is not a failure of behaviour.
  2. Match to guidance. For each, find the rule that existed at the time and says what the test checks. Exclude a task if the rule was later edited from that bug, or the test checks things the rule never says.
  3. Write behaviour graders. Drive the code through the interface it had when frozen. Assert what the rule asks, and add a positive control so that refusing everything fails. Write each grader against the frozen and fixed states only, and save its check (fails before, passes after) as a receipt.
  4. Build the room. A copy at the frozen commit, with one commit of history and no agent configuration. Limit the agent’s shell to running tests and reading git state.
  5. Pre-register. Commit the arms, the measures, the bar for each reading and what each outcome would mean, before any counted session.
  6. Run. Interleave all arms in one seeded order with at least five repeats. Save each session’s patch and event stream. Count a session that returns nothing as missing, not as a fail.
  7. Grade from patches. Apply each patch to the frozen code, overlay the hidden grader and run it. Report patches that did not apply.
  8. Publish receipts. Commit the result file, stamped with the commit and script hash. Add a test that fails if a quoted figure disagrees with it.

Measuring choosing. Use the replay’s tasks, where the right rule is known, or label a larger set of past changes. Rebuild the store as of each change. Report recall at the size your review can afford to check one at a time, with a paired bootstrap over changes.

Scripts (Python, in the public repository):

ScriptDoes
outcome_replay.pyBuilds rooms, runs arms in seeded order, saves patches
outcome_regrade.pyRe-grades saved patches under new graders
replay_graders_4aw.pyThe behaviour graders, with positive controls
replay_list.py, replay_one.pyThe list and one-at-a-time arms
review_e2e.pyEnd-to-end review; counts sub-agents from the event stream
plan_reach.py, when_lines.pyChoosing from task, plan and diff; applies-when lines
conditions_agree.pyTwo-judge agreement with Cohen’s kappa
realwork_reach.pyRebuilds the lesson store as of a commit
run_guard.pyStamps receipts with commit, dirty state and script hash

Also: pick_fix.py (choosing from the code, §4bc), briefed_follow.py (§4as), explain_search.py (§4at) and default_read.py (§4au).

The pre-registrations and results are in evals/REPORT.md, sections 4as to 4bc, and the receipts in evals/results.

Costs observed (API-priced; a subscription plan may pay none of it):

RunPer sessionMedian time
One agent session, one lesson$0.1272 s
One agent session, list of ten$0.1383 s
Ten lessons one at a time, a fresh session each$0.81453 s
End-to-end review with sub-agents$0.73214 s
One judge verdict, Fable 5.1about $0.76

Pitfalls worth a check in your own harness. A usage limit looks like an empty reply, so count empty replies and stop after a run of them. Shared model objects may not be thread-safe. Long diffs get cut, and the defect may lie past the cut. An eval that calls internal functions can skip the configuration users run.

How it improves over time, and what is next

The system improves through three loops. Two run in production: keeping lessons true to the code, and learning as the team builds. The third, the evaluation loop in this document, tests okl’s first job: whether lessons reach the agent and change its work.

Keeping lessons true to the code (in production). A lesson that names the files it governs and a check that proves it is verified by running that check. When those files change, the push gate or CI’s drift gate goes red until the check runs again, and the briefing marks the lesson stale. A lesson with no check behind it is shown as unchecked. Today this covers only lessons that name files and carry a check: 16 of 155 at the 2026-10-31 census. When patches were written to break lessons, drift caught 14 of the 36 a judge confirmed as breaks (§4af), and 16 of 24 after a fix, on the lessons whose check runs from the commit (§4ah).

Learning as the team builds (in production). When a session ends, a hook asks what was learned, and the answer is recorded as a lesson with a scope and tags. Decisions, conventions and fixes from today’s work brief tomorrow’s session.

The evaluation loop (this document). Each experiment asks one question with a bar fixed in advance. Its result points to one change in okl, and that change is measured again on the same tasks, against a control in the same sitting. A lesson the measurement misses is logged, so the next method is tested on the cases the last one failed. The diagram under “How the work iterates” draws this chain from the first observation to today.

Open questions, in the order I plan to test them.

  1. Can okl choose from the code? §4bc tests choosing by hunk, by a one-sentence description of each hunk, by exact code names, and by screening a pool of 30 one lesson at a time. It needs the screened set to hold at least 0.15 more of the broken lessons than today’s pick, at 12 lessons or fewer. It did not: the screened set beat the pick by 0.11, short of the bar, at 22 lessons, because the screen said yes to most lessons. Next: a screen that asks the review’s own question (does this change break this lesson?), or the fused list cut to ten.
  2. A cheaper shape for one at a time. Ten fresh sessions cost about seven times one. Untested: several lessons per fresh session, or only the few a screen keeps.
  3. Does the briefing’s size matter? A seeded test of the start-of-task briefing: the full tiered list, the top five, or the top one.
  4. Do agents finish the review? A trial session, discarded before counting, stopped after three of its ten sub-agents. How often that happens, and whether wording changes it, is unmeasured.
  5. Beyond one repository. The same replay on other projects’ histories, with keys audited by people.
  6. Does it compound? Whether a store makes an agent better over months, not just on one change, needs a longitudinal design I have not built. Errors in stored experience can compound too (Xiong et al.), so it would have to measure harm as well as help.

References

Papers on agents and judges were read at the source, at the sections noted in the repository’s research notes. The three methods papers are standard references.

  1. Ahmed, T., Devanbu, P., Treude, C., Pradel, M. Can LLMs Replace Manual Annotation of Software Engineering Artifacts? MSR 2025. arXiv:2408.05534.
  2. Gloaguen, T., Mündler-Sasahara, N., Müller, M. N., Raychev, V., Vechev, M. Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? Preprint, arXiv:2602.11988v3, 2026.
  3. Lee, Y., Kim, J., Kim, J., Cho, H., Kang, J., Kang, P., Kim, N. CheckEval: A Reliable LLM-as-a-Judge Framework for Evaluating Text Generation Using Checklists. EMNLP 2025. arXiv:2403.18771.
  4. Li, X., Liu, Y., Chen, W., You, B., et al. SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks. Preprint, arXiv:2602.12670v4, 2026.
  5. Pombal, J., Rei, R., Martins, A. Self-Preference Bias in Rubric-Based Evaluation of Large Language Models. COLM 2026. arXiv:2604.06996.
  6. Tsuchida, Fan, Shimari, Kula, Matsumoto. Do Coding Agents Follow AGENTS.md? The Effect of Policy Wording on Instruction Compliance. arXiv:2611.04367, 2026.
  7. Xiong, Z., Lin, Y., Xie, W., He, P., Liu, Z., Tang, J., Lakkaraju, H., Xiang, Z. How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behavior. Preprint, arXiv:2505.16067v2, 2025.
  8. Xu, Wu. Stale Instruction Files Override the Code in Coding Agents: A One-Sentence Fix. Preprint, arXiv:2610.30571, 2026.

Methods:

  1. Cohen, J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1), 1960. (Kappa.)
  2. Cormack, G. V., Clarke, C. L. A., Büttcher, S. Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. SIGIR 2009. (Fusion.)
  3. Efron, B. Bootstrap methods: another look at the jackknife. Annals of Statistics 7(1), 1979. (Intervals.)

Primary record: okl’s evaluation report, sections 4as to 4bc, with every receipt in evals/results.