A methods paper · 12 November 2026 · the primary record is okl's evaluation report
On five real defects, a coding agent given no lesson fixed none, and one given the right lesson fixed about two in three. Given the same lesson in a list of ten, it fixed about one in seven.
I study okl, which keeps a coding agent true to a team’s rules, specs and docs, and keeps those true to the code. okl calls each thing it holds a lesson: a rule, decision, convention or doc claim, some written as policy and some after a defect. This study tests the first half of that claim. When a lesson applies to a change, does putting it in front of the agent change the code it writes, and what limits that?
The main method is an outcome replay. I take real defects from the project’s history and restart each one from the change as first written, before review caught the flaw. Defects are the test bed, not the point: they are where a lesson’s effect can be graded, because the real fix supplies a test. An agent finishes the change under a controlled condition. A hidden test, derived from the real fix, grades the result by its behaviour. Three of the five graders have a positive control, so a fix that refuses everything fails them.
In pre-registered pilots (Sonnet as the agent; 25 sessions or picks per arm, 75 to 150 per experiment):
Separately, two judge models agreed on whether a change broke a lesson with Cohen’s kappa 0.742. Appending yes/no conditions to each lesson did not raise agreement (0.599; change −0.143 [−0.368, 0.094]).
The tasks are few, from one repository, and chosen because a lesson applies. These are pilots that decide what to test next, not effect sizes to generalise. I describe each method in enough detail to rerun it on another project’s history, with the integrity rules that kept the results honest.
A written lesson works on an AI coding agent much as a protocol note works on a new lab member. Read alone and at the right moment, it changes what they do. Buried in a binder of ten, it is mostly skimmed.
The setting. Coding agents are language models that edit a codebase: they read files, write code and run the tests. They start each session knowing nothing of a team’s rules, specs, architecture decisions or docs, and nothing keeps those in step with the code. okl does both jobs. It holds what the team maintains as lessons: a rule, a decision, a convention, a doc’s claim, or a warning (“a gate that checks files must read the committed files, not the working copy”). It puts the ones that apply in front of the agent, and it flags when the code moves away from what one describes. Recording what each session learns comes third. A briefing goes in before the work starts, chosen by matching the task’s words. A review runs after the code is written, with the agent checking its change against lessons one at a time.
The experiment. I test the first job: does a lesson in front of the agent change its code? Past defects make good cases because the right answer is known: a lesson covered the defect, and its real fix added a test. Each is treated like a case in a controlled trial:
The plan, the conditions and the pass bar are written down and committed before any session runs. That makes the study pre-registered: any later change to the plan is a dated amendment, written before the data it affects is seen.
What I found. The lesson itself works: with it, the agent fixed most defects; without it, none. How the lesson is handed over matters as much as its content: a list of ten cut the effect by two thirds or more, and asking about each lesson separately restored it. The hardest remaining step is choosing which lessons to show. The best methods I tried find the right one about half the time, and most find less.
How confident to be. Moderately, about direction; not about size. There are five tasks from one project, one agent model, and five repeats per condition. The results decide what to test next. They are not a general law about AI agents.
Whether written guidance changes a coding agent’s work is an open empirical question, and recent studies disagree on the answer.
On repository instruction files, Gloaguen et al. found that context files do not generally raise task success and add over 20% to cost. Xu and Wu found the opposite failure: agents followed a stale file over the code in 90.1% of choices. One added sentence moved code-following from 4.2% to 91.4%. SkillsBench found curated skills raised average pass rates from 33.9% to 50.5%. Skills the agent wrote for itself scored below none. Tsuchida et al. found that conditional wording (“if X, do Y”) was followed less than prescriptive or explanatory wording.
Reach is not compliance. okl’s first evidence was a behaviour result on authored tasks: on eight tasks written to invite defects the store already knew about, the briefing cut defect reproduction from 42% to 7% (68 of 163 against 11 of 164 attempts, seven runs, graded by a model judge). Later studies on real pull requests measured reach: whether the right lesson appeared in the briefing. On 58 of them, lessons the briefing showed were followed no more often than lessons it did not: 63.3% against 72.3%, a difference of −9.0 points [−22.0, 4.2]. That observational result is a correlation and could hide an effect. It still meant that reach alone could not be the measure.
Model-made answer keys were not enough. The real-work studies graded “was this lesson followed?” with a judge model. A third model, asked to settle 24 disputed cases, did not: it called only 2 of the 4 agreed-broken controls broken. Agreement between models predicts whether a labelling task suits them at all (Ahmed et al.), and judges favour their own output (Pombal et al.). So I needed outcomes that do not rest on a model’s opinion: a test that passes or fails. A later blind audit bore this out: a second model agreed with only a third of the pairs the key called broken, and no person has yet labelled them, so the results scored against that key (§4bc to §4be) are scored against a model’s key (REPORT §4bh).
The three questions. Each experiment below answers one:
The first two are about compliance and the third is about retrieval. The results show that they fail in different ways and need different fixes.
okl does three jobs, in this order: it keeps a coding agent true to a team’s rules, specs and docs; it keeps those true to the code; and it keeps learning as the team builds. It is local and git-native, and runs no generative model by default. These experiments test the first job. Most hand the lesson text to the agent directly, to isolate the effect of the lesson itself; one (§4az) runs okl’s own review after writing, and one (§4as) observes the briefing on real work. The read-time briefing was not tested. All of them use one repository, okl’s own, with about 300 lessons.
| Term | What it is | Where it acts |
|---|---|---|
| Lesson | One record of something the team maintains: a rule, decision, convention, doc claim or warning. It has a title, the rule, the fix, why, tags, and optionally the files it governs and the check that proves it. Some are named for the incident behind them rather than the code they govern (§4bb). | The store, .okl/okl.db (SQLite), plus shared packs |
| Briefing | Lessons chosen by matching the task’s words. Full-text search fused with an embedding search; up to 5 shown in full and 12 more in one line each, plus up to 10 standing lessons. | Before the agent starts (a prompt hook) |
| Read-time briefing | Lessons whose governed files match a file the agent opens. | When the agent reads a file |
| Review after writing | Five steps in okl check --change: (1) say what the changed code does, one mechanism a sentence; (2) pick the ten lessons the change most likely breaks, from every lesson’s one line, those governing a touched file first; (3) check each in its own fresh sub-agent with one question: does this change break this lesson? (4) without sub-agents, one lesson per pass; (5) report. | After the code is written; opt-in |
| Drift gate | A lesson with governed files and a recorded check goes red when those files change after the check last ran. | In CI |
Two definitions recur. A lesson is broken by a change when the change does what the lesson warns against, or fails to do what it requires. A lesson is found when it appears in the list a method returns, at the stated size.
Terms for readers outside the field.
Nothing in the methods depends on okl’s storage. Any system that holds written lessons and decides which to show an agent can be tested the same way.
The technique is a loop: measure one thing against a bar set in advance, change one thing because of the result, and measure again against a control in the same sitting.
Read it in zigzag. Each box on the left is a pre-registered experiment and its result. The box beside it is the change that result caused, and the arrow down is the next experiment, which tests that change. Two of the changes were to the measurement itself (the controlled replay and behaviour graders). The rest were to okl, or to the question asked next.
Three rules keep the loop honest. A change is never judged on the data that suggested it. A failed bar is reported as failed, and the change it would have justified is not built. Each new experiment re-runs the earlier arm it is compared with, so a result is judged against a control from the same sitting, not a number from another day.
The outcome replay measures compliance by behaviour: a real defect is restarted just before review caught it, an agent finishes the change under one condition, and a hidden test from the real fix grades what it leaves.
1. Choose tasks from real history. A candidate is a past change whose real fix added a test that fails on the change as first written and passes after the fix. Read each test against the lessons that existed when the change was made. Keep a task only when the test checks what a lesson already said. Exclude for cause:
In okl’s history five tasks survived: #115, #168, #196, #198 and #309.
2. Freeze the start. Each session starts from the change as first written, before review found the defect. It runs in a copy with one commit of history and no agent configuration files. This tests an agent finishing a written change. Starting from nothing would grade the private names the real author happened to choose.
3. Apply one condition per arm. Every arm receives the same instruction: get this change ready to merge. Arms differ only in what lesson text the agent sees and how. The conditions used so far:
| Experiment | Arms |
|---|---|
| Lesson vs none, and its wording | no lesson; the lesson as stored; as a contract (when, do, done when, does not apply, why); contract plus examples |
| Lists | alone or among nine others; told it applies or not (2 × 2) |
| One at a time | alone; list of ten; the same ten, each in its own fresh session |
| End to end | no review; old review (one pass over ten); new review (a sub-agent per lesson), with the agent choosing |
The nine other lessons in a list are those okl’s search ranks highest for the task’s words, excluding the right one and any lesson that cites the task. A separate agent wrote the contracts and examples from the lessons alone, never shown the tasks.
4. Run the agent. Sonnet, one session per run, with tools to read, search and edit files and to run only pytest, git diff and git status. Five repeats per arm and task, in a seeded random order, so a run that stops early favours no arm. In the one-at-a-time arms a run is one fresh session per lesson, so each of their runs is about ten sessions, not one. Each session saves its patch.
5. Grade with a hidden behaviour test. After the session, the grader is laid over the tree it left. The agent never sees it. A session passes when every case in the grader passes. Each grader must fail on the frozen code and pass on the real fix, and that check is itself saved as a receipt.
Write behaviour graders, not the fix’s own test. Four of the five real tests called a function the fix invented. On the frozen code they failed on a missing name, so no agent could pass without guessing it. The replacement graders drive the code through the interface it had when frozen and assert what the lesson asks. For example, two commands that resolve to the same model must not start a run. The graders for #168, #198 and #309 each have a positive control, so a fix that refuses everything fails. #196’s accepts a refusal, which also keeps its lesson, and #115 keeps the real fix’s own test, which fails by assertion.
6. Grade from saved patches. Because each session saves its patch, graders can be fixed and re-run without re-running agents. The regrader reports any patch that did not apply, was cut off, or left a file outside the patch.
Measures. The pass rate for an arm is averaged over tasks, so each task weighs the same whatever its repeats:
pass(a) = 1/|T| · Σ_{t ∈ T} #{passing sessions of arm a on task t}
/ #{sessions of arm a on task t}
Also reported: sessions that edited anything, harm (another test in the grader’s file that passed on the frozen code and now fails), turns, time and API-priced cost. In end-to-end runs, sub-agent use is counted from the session’s own event stream.
Choosing is measured without running an agent: a method returns a ranked list of lessons for a change, and I count how often the lessons that change needed are in it. This is cheap, so it can run on many more changes than the replay.
Two benchmarks.
Lessons as they stood. Every method searches only the lessons that existed when the change was frozen, so a lesson written later cannot leak the answer. Their text, though, comes from a later snapshot, and some were extended from these same changes, which can help every method alike (see Threats).
Recall at k. For change c with broken lessons B(c), and a method’s top-k list L_k(c):
recall@k = 1/|C| · Σ_{c ∈ C} |L_k(c) ∩ B(c)| / |B(c)|
k = 10 is the review’s own size: the review then checks each chosen lesson one at a time.
Starting points and methods tested.
| Method | Reads | Chooses by |
|---|---|---|
| Task | the task’s title and issue | okl’s search (full text fused with an embedding); or a model picking ten from every lesson’s one line |
| Plan | a plan a model writes in the repository before any code, one mechanism per sentence | a model picking ten |
| Diff | the finished change, cut to 40,000 characters | a model picking ten (today’s review) |
| Explanation | a model’s account of what the changed code does | okl’s search |
| Applies-when lines | each lesson’s line plus a model-written “applies when” by mechanism | a model picking ten |
| Chunk | each changed hunk (the 20 largest, 1,500 characters each) as its own query | okl’s search, fused |
| Description | one model-written sentence per hunk saying what it does | okl’s search, fused |
| Anchor | code names, flags and paths the change uses; files lessons govern | exact matching, rare names weighted up; no model |
| Screen | the model’s pick fused by rank with the chunk, description and anchor lists, top 30 | a small model (Haiku), asked of each lesson alone: does this change do something this lesson is about? |
Fusion. Several ranked lists are merged by reciprocal-rank fusion, with the usual constant 60:
score(ℓ) = Σ_L 1 / (60 + rank_L(ℓ))
Uncertainty. Differences between methods are paired by change and bootstrapped over changes, 2,000 draws with a fixed seed. The 95% interval is reported.
Every rule below exists because an evaluation of model behaviour has more free choices than a reader can see. Each one closes a specific way to fool myself, and several were added after one did.
| Practice | What it prevents | How it is held here |
|---|---|---|
| Pre-registration | Choosing the measure or the bar after seeing the data | Arms, measures, decision rule, threats and budget are committed to the report before any counted call |
| Dated amendments | Silent changes mid-study | A change after registration is written, dated and committed before the data it affects is seen, saying what was already seen |
| A rule with a reading per outcome | Reading a null as a near-miss | Each section states in advance what each outcome means (“collapses in a list”, “stays opt-in”) and what work it points to |
| Same-sitting controls | Drift in models and tools between days | Every bar is stated against a control arm re-run in the same sitting, never against an old number |
| Development and test halves | Tuning on the data that decides | Prompts are revised on the development half only, at most three times. The test half runs once. |
| Freeze by hash | Editing inputs after seeing outputs | Conditions and lines are hashed in the script; the test half refuses to run under an unfrozen hash |
| Judge is not the generator | Self-preference in model judges | The harness refuses a run whose judge model is the generator; agreement is measured between two models that wrote neither the lessons nor the key |
| Blind authorship | Leaking the answer through the treatment | Contracts, examples and applies-when lines are written by a separate agent shown only the lesson text, never a task |
| Graders with positive controls | A grader that a do-nothing or refuse-everything fix passes | Each grader fails on the frozen code and passes on the real fix, a check saved as a receipt; three of the five also have a case that refusal fails |
| Seeded call order | Stopping early favouring one arm | All arms interleave in one seeded random order |
| Completeness thresholds | Selective stopping | A reading needs about 90% of sessions complete, a count fixed in advance; a run cut by a usage limit resumes at the same commit and prompt, and partial receipts are deleted |
| Run guard | Results that no commit can reproduce | Each receipt carries the commit, whether the tree was dirty and the script’s SHA-256 |
| Receipts and figure tests | Transcription errors and stale numbers | Every figure the report quotes has a committed receipt, and a test fails if the report’s text and the receipt disagree |
| Labelled interpretation | Post-hoc stories passing as findings | Readings not fixed in advance are headed “What it means, not pre-registered” |
The protocol has caught me several times, each correction recorded in the report beside the claim it changed. A grader filter read an empty field and excluded nothing (§4aw). A briefing size was quoted as 12 places when it is up to 17 (§4ax). An agreement figure came from cases chosen because they were disputed (§4au, corrected from §4av). A script would have read plans from the working tree, not the committed receipt (§4bb). A thread fault stopped two runs before any row was kept (§4bc).
The lesson itself changes the code, how it is handed over decides whether it does, and choosing the right lesson is the largest gap found.
Presentation (§4aw to §4az). Each dot is one arm of 25 runs (a run is one session, or one per lesson in the one-at-a-time rows); a row’s spread mixes arms (told it applies or not) and sittings.
With no lesson, the agent left all 25 changes as it found them and called them ready. In a list it edited in 16 of 50 sessions, against 39 of 50 alone. Telling it the lesson “bears on this change” made no difference, alone or in a list. One at a time cost $0.81 a session against $0.12 alone. Its median session made one edit for the nine other lessons (whether those edits helped was not judged), and no session in any arm broke another test. Rewording the lesson changed little: as a contract it scored 0.52 and with examples 0.6, against 0.64 as stored.
Choosing (§4ba, §4bb). A picker read every lesson’s one line and named ten, after reading the task, a plan written before any code, or the finished diff.
The left half and right half are separate sittings. In §4bb’s own sitting the plain list scored 0.36, 0.44 and 0.4, so the lines added 0.1 on average from a plan or diff, under the 0.15 bar. #196’s lesson is written about audits of files, and the faulty code was an eval script loading a folder. On #309 no plan named the code that resolves the store, and in the diff it lay past the 40,000-character cut.
On the 34 development-half changes with a broken lesson, okl’s search found 14.3% of broken lessons in its top five from the task’s words. It found 12.9% from the added code and 14.3% from a model’s explanation of it. A model reading that explanation and picking ten from every lesson’s line found 24.3% at five and 40.0% at ten (§4at; development half only, in-sample and not pre-registered).
Judge agreement (§4av). On 70 cases, Sonnet and Fable 5.1, each reading one lesson against one change, agreed on broken-or-not in 62 cases (kappa 0.742). With yes/no conditions added they agreed on 56 (kappa 0.599), a change of −0.143 [−0.368, 0.094].
Choosing from the code (§4bc). Pre-set reading: choosing does not improve enough. Each list was scored by the share of a change’s broken lessons it held, on the 58 keyed past changes, as a model’s key labelled them (see the audit above). The pool fuses the model’s pick with the three lists drawn from the code, and a small model (Haiku) screened each pooled lesson.
The gain’s 95% interval was [0.04, 0.191]. The pool holds the pick’s own list, so part of that gain is by construction, and it came at 22.1 lessons on average against the 12 the pre-registration allowed. The screen said yes to 1,281 of 1,727 lessons. Each list built without the pick held less than the pick did. Every method still missed all of a change’s broken lessons on some changes: the pick on 19 of 58, the pool and the screen on 13.
Every experiment, with its receipt (in evals/results/):
| Section | Question | Design | Reading (pre-set unless noted) | Receipt |
|---|---|---|---|---|
| 4as | Are lessons the briefing showed followed more often? | Observational, 58 PRs, 317 pairs | No association (exploratory, not pre-registered) | briefed-follow-20261110-0426.json |
| 4at | Does explaining the code find broken lessons? | 34 development changes | No better than the task’s words (in-sample; the test half was not run) | explain-search-dev-20261110-1306.json |
| 4au | Should the review be on by default? | 20 unseen changes | Rule fails; off by default | default-read-20261110-1438.json |
| 4av | Do yes/no conditions raise judge agreement? | 70 cases, 280 verdicts | Rule fails; not shipped | conditions-agree-test-20261111-0038.json |
| 4aw | Does a lesson change the code, and does wording matter? | Replay, 4 arms, 100 sessions | No lead for the contract; the lesson’s own effect was reported with no rule | outcome-replay-20261111-0129.json |
| 4ax | Does it survive a list of ten? | Replay, 2 × 2, 100 sessions | Collapses in a list | replay-list-20261111-0249.json |
| 4ay | Does one at a time recover it? | Replay, 3 arms, 75 sessions | Works | replay-one-20261111-1355.json |
| 4az | Does the rebuilt review work end to end? | Replay, 3 arms, 75 sessions | Stays opt-in | review-e2e-20261111-1543.json |
| 4ba | Does a plan find the lesson? | 75 picks | Not well enough | plan-reach-20261111-1435.json |
| 4bb | Do applies-when lines help a picker? | 150 picks | Not enough | when-lines-20261112-0058.json |
| 4bc | Can okl choose from the code? | 58 changes | Does not improve enough | pick-fix-20261112-0425.json |
Most ideas failed their pre-set bar, and the failures shaped the method as much as the successes did. Each row was decided by a rule written before the data.
| Attempt | Expected | What happened | What it taught |
|---|---|---|---|
| Review on by default (§4au) | It finds at least 30% of broken lessons | 17.4% [4.0, 37.5]. It picked half the broken lessons, then judged most of them followed | Finding a lesson and checking it are separate steps; the check reading ten at once was the weak one |
| A third model to settle disputes (§4au) | It agrees with the control cases both sides called broken | 2 of 4 controls | Disputed cases are the wrong sample for agreement; a key needs an outcome, not another opinion |
| Yes/no conditions on lessons (§4av, after CheckEval) | Agreement between judges rises by 0.15 | Kappa 0.742 without, 0.599 with: a change of −0.143 [−0.368, 0.094]. Broken calls rose from 46 to 74; false alarms from 1 to 4 | The premise was wrong: two judges reading one lesson at a time already agreed. Conditions made them stricter, not more consistent |
| Lessons as contracts (§4aw) | Contracts beat the stored text by 0.2 | 0.52 against 0.64 | Wording was not the bottleneck. A “does not apply” clause may give the agent a reason to stop |
| The real fix’s test as grader (§4aw) | It grades fairly | 4 of 5 called a function the fix invented, so they failed on a missing name. The filter meant to catch that read an empty field | Read the actual exception on the frozen code. Write behaviour graders with positive controls |
| A plan to choose lessons early (§4ba) | At least 0.6, and 0.25 above the task’s words | 0.48, and 0.12 above | A plan chooses as well as the finished change, so choosing can move earlier, but neither is good enough |
| “Applies when” lines by mechanism (§4bb) | Hit rate up by 0.15 | Up by 0.1, nearly all on one task | A line written from the lesson alone keeps the lesson’s own framing, and cannot know which other code shares its mechanism |
| The size of a default briefing (§4ax threats) | Quoted as 12 places, with the right lesson’s search ranks | A briefing is up to 17 places (5 in full, 12 in one line), and the ranks came before the briefing’s own filters. Run as shipped, it showed the right lesson on 2 of 5 tasks. Corrected by amendment before scoring | Measure what the product shows, not an intermediate list |
| §4bc’s first two runs | Rows kept | None. The embedding’s tokenizer cannot be shared across threads | Shared model objects may not be thread-safe; count empty results and stop on a run of them |
§4bc joins this list. Fusing today’s pick with lists drawn from the code, then screening each lesson, beat the pick alone, but by less than its bar and at twice the size, because the screen asked whether a change bears on a lesson and said yes to most. A screen has to tell lessons apart, not just find related ones.
The successes were fewer. The lesson itself changes the code (§4aw). One at a time recovers what a list loses (§4ay). The review rebuilt on that beats the old one end to end (§4az).
A rule changes an agent’s code when it reaches the agent on its own, at the moment it applies, and still matches the code. Length and wording mattered less than that in every test here. These come from five tasks in one repository, so treat them as hypotheses to test on your own history with the replay.
One common habit was not tested here: adding a line to the file after every mistake. By points 1 and 7, a file grown that way gets longer and goes stale, so pair it with pruning and checks.
The largest threat is size: five tasks from one repository, chosen because a lesson applies, run with one agent model.
Internal validity.
Measurement validity.
External validity.
What these results do not show. They do not show that lessons improve code quality in general, help with mistakes the store has never seen, or compound over months. They do not show effects on human developers, or on other languages, repositories or agents.
The replay needs three things: a repository whose fixes added tests, written guidance with dates, and the ability to rebuild that guidance as it stood at any commit. A first pilot of 100 agent sessions cost about $12 at API prices, a median of 77 seconds a session.
Outcome replay, step by step.
Measuring choosing. Use the replay’s tasks, where the right rule is known, or label a larger set of past changes. Rebuild the store as of each change. Report recall at the size your review can afford to check one at a time, with a paired bootstrap over changes.
Scripts (Python, in the public repository):
| Script | Does |
|---|---|
| outcome_replay.py | Builds rooms, runs arms in seeded order, saves patches |
| outcome_regrade.py | Re-grades saved patches under new graders |
| replay_graders_4aw.py | The behaviour graders, with positive controls |
| replay_list.py, replay_one.py | The list and one-at-a-time arms |
| review_e2e.py | End-to-end review; counts sub-agents from the event stream |
| plan_reach.py, when_lines.py | Choosing from task, plan and diff; applies-when lines |
| conditions_agree.py | Two-judge agreement with Cohen’s kappa |
| realwork_reach.py | Rebuilds the lesson store as of a commit |
| run_guard.py | Stamps receipts with commit, dirty state and script hash |
Also: pick_fix.py (choosing from the code, §4bc), briefed_follow.py (§4as), explain_search.py (§4at) and default_read.py (§4au).
The pre-registrations and results are in evals/REPORT.md, sections 4as to 4bc, and the receipts in evals/results.
Costs observed (API-priced; a subscription plan may pay none of it):
| Run | Per session | Median time |
|---|---|---|
| One agent session, one lesson | $0.12 | 72 s |
| One agent session, list of ten | $0.13 | 83 s |
| Ten lessons one at a time, a fresh session each | $0.81 | 453 s |
| End-to-end review with sub-agents | $0.73 | 214 s |
| One judge verdict, Fable 5.1 | about $0.76 |
Pitfalls worth a check in your own harness. A usage limit looks like an empty reply, so count empty replies and stop after a run of them. Shared model objects may not be thread-safe. Long diffs get cut, and the defect may lie past the cut. An eval that calls internal functions can skip the configuration users run.
The system improves through three loops. Two run in production: keeping lessons true to the code, and learning as the team builds. The third, the evaluation loop in this document, tests okl’s first job: whether lessons reach the agent and change its work.
Keeping lessons true to the code (in production). A lesson that names the files it governs and a check that proves it is verified by running that check. When those files change, the push gate or CI’s drift gate goes red until the check runs again, and the briefing marks the lesson stale. A lesson with no check behind it is shown as unchecked. Today this covers only lessons that name files and carry a check: 16 of 155 at the 2026-10-31 census. When patches were written to break lessons, drift caught 14 of the 36 a judge confirmed as breaks (§4af), and 16 of 24 after a fix, on the lessons whose check runs from the commit (§4ah).
Learning as the team builds (in production). When a session ends, a hook asks what was learned, and the answer is recorded as a lesson with a scope and tags. Decisions, conventions and fixes from today’s work brief tomorrow’s session.
The evaluation loop (this document). Each experiment asks one question with a bar fixed in advance. Its result points to one change in okl, and that change is measured again on the same tasks, against a control in the same sitting. A lesson the measurement misses is logged, so the next method is tested on the cases the last one failed. The diagram under “How the work iterates” draws this chain from the first observation to today.
Open questions, in the order I plan to test them.
Papers on agents and judges were read at the source, at the sections noted in the repository’s research notes. The three methods papers are standard references.
Methods:
Primary record: okl’s evaluation report, sections 4as to 4bc, with every receipt in evals/results.