The business caseThe problem this solves

'Which dealers have duplicate warranty claims' is a SELECT and nobody needs a language model for it -- and it hands an investigator the same dealer at the top of the list every Monday, because a claim that was countable last week is still countable this week. The question a warranty audit programme actually has to answer is the next one: of everything in this book, what was NOT countable the last time anybody looked, and is this dealer escalating or closing out? Three things make that unanswerable from the page in front of you. A claim can become countable with nothing about it having changed, because a clock crossed. A dealer's level depends on how many consecutive runs it has already been at REVIEW. And a finding an investigator declined last week may or may not have taken its claims out of the population, depending on a sentence somebody typed. Somebody working a warranty audit queue by hand every Monday: pulling each dealer's open claim book, working out which of the duplicates, repeat repairs, inflated labour lines and out-of-population recall claims were already raised weeks ago, reading the investigator's notes to see which findings were declined and whether the reason was a real one, and only then deciding whether this dealer is getting worse or quietly closing out.

The inputThe actual dealer readings

The corpus is 160 dealer readings, 1.00 MB (json 2 · jsonl 2 · txt 160). A WARRANTY FRAUD QUEUE IS A LIST OF NAMED DEALERS WITH AN ALLEGATION BESIDE EACH ONE. There is no public corpus of that and there never will be, for the same reason there is no public corpus of anybody's internal audit file: publishing it IS the disclosure, and a dealer principal is an identifiable person against whom nothing has yet been proven. Generating it also bought the one thing a captured corpus cannot give: the answer key is src/programme.replay's output over the RENDERED pages, so a set-and-interval rule this fiddly cannot carry its author's misreading into the score -- and evals/check_labels.py re-derives all 160 rows from the shipped files before any run may spend. It also bought the nine storylines. A captured book contains whatever it contains; this one deliberately separates a dealer whose problem is growing from one whose identical-looking pile is three weeks old and closing, because those two are indistinguishable on a single page and are the entire reason this kit carries state. ⚠︎ AND IT SHIPS BLOCKED PENDING ANCHOR. The 30-day duplicate span, the 180-day repeat span, the 1.5x labour multiple, the 21-day parts clock, the 500 USD parts floor, the 30-day chargeback window, the four levels and the three-consecutive-runs rule are INVENTED for this kit. They reproduce no manufacturer's warranty policy, no dealer agreement, no regulator's guidance and no industry code, and name none. The catalogue row this kit was built from records the guardrail as BLOCKED-PENDING-ANCHOR.

Synthetic Record
------------------------------------------------------------------------------
This file is synthetic. It was generated for an open evaluation kit and describes no real
dealer, person, vehicle, warranty claim, campaign or manufacturer. Dealer DLR-0001, run W1 (2026-07-06).

Dealer
------------------------------------------------------------------------------
Dealer code    : DLR-0001
Region         : North
Franchise      : volume marque
Claim volume   : small (under 400 claims a year)
Enrolled       : 2018-01-01
Run date       : 2026-07-06
Previous run   : none
Claims in book : 4 (every open claim paid in the last 90 days, re-read whole this run)

Audit Programme Rules
------------------------------------------------------------------------------
Countable claims. A claim in the book below is COUNTABLE against one of five patterns:
  DUPLICATE_CLAIM          two or more paid claims on the same VIN for the same failure code
                           within 30 days. The later claim of each pair is countable.
  REPEAT_REPAIR            3 or more paid claims on the same VIN for the same component
                           group within 180 days. The third and each later claim is countable.
  LABOUR_TIME_INFLATION    labour hours claimed above 1.5 times the published time.
  PARTS_NOT_RETURNED       a claim above 500.00 USD recording parts returned N, more than
                           21 days after payment. It becomes countable on the run that
                           crosses that clock, not on the run the claim first appeared.
  OUT_OF_POPULATION_RECALL a claim citing a campaign whose VIN sequence falls outside the
                           campaign's affected range, or whose build date falls outside the

The outcomeWhat a good result looks like

Per dealer per run: the level (NONE / WATCH / REVIEW / INVESTIGATE), the pattern attributed to it, how many claims became countable since the previous run, and the movement against the level last reported -- plus a four-field carried position the next run is judged against, written by code from the parsed claim book and never from the model's reply.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Running a weekly duplicate-and-pattern watch over a dealer network at all — the free carry-aware rule floor first, at $0.00 -- then add the model only if you have measured the margin on your own book
  98.12 pct on the level, 100.0 pct on the pattern, 96.88 pct on the count, 98.12 pct on the movement, 0 missed raises and 3 false alarms in 103 quiet readings -- with no model, no key and no network. It is evals/baseline.py and you can run it in a fresh clone right now. The model reaches 98.75 pct on the level for 0.014193 USD a reading; that margin is +0.63 points.
• Deciding whether an investigator's declination note removes claims from the count — the model, on that subset only
  This is the one clause a rule cannot express, and it is the only place the model earns its price on this kit. It reads 22 of the 24 prose-dependent readings against the free carry-aware floor's 21 -- 91.67 pct against 87.5. The keyword table classifies 12 of 16 wordings and the four it misses are substantive reasons written without the obvious word. EVERY error in both best arms lives here.
• Choosing how often to run it — weekly, on the shipped constants -- and re-derive it for your own
  evals/missed_run.py settles it arithmetically and for free. The parts clock is 21 days and the chargeback window 30, so a countable claim has 9 days of headroom; a weekly run is inside it and a fortnightly one is not. One missed run costs 6 claims worth 9,779.29 USD on this corpus, and changes 32 of 40 run-4 answers.
• Standing a monitor up for the first time on a book nobody has watched — a person, working the first run's output as a backlog rather than as a queue
  A monitor's first run has no previous reading, so every standing claim reads as new. On this corpus run 1 surfaces 20 claims worth 30,380.79 USD that were ALREADY past the chargeback window on the day they were found. That is not a false positive and it is not a model failure -- it is a backlog, and treating it as a weekly queue would put a network's worth of unactionable findings in front of an investigator on day one.
• Debiting a dealer, raising a chargeback or opening an audit — the audit manager, on evidence an investigator has worked
  Nothing in this kit does any of those and nothing should be added that does. The cap on the catalogue row is money-movement, hard and non-configurable, and evals/check_labels.py asserts the absence of ten money-moving call names before any run may spend.

And where nothing here is good enough:

• Running the watch with no memory between runs -- a weekly query, joined to nothing — nothing here. Do not do it
  The free one-reading floor is exactly that, and it is what most programmes actually run. Level 51.88 pct, 44 false alarms in 103 quiet readings, a structural 0.0 pct on the counts that need memory, and UNCHANGED as the answer to 'what moved' on every reading of every week. It surfaces every real raise and buries them in 44 spurious ones.
• A dealer network with years of weekly history, or thousands of dealers — nothing here yet -- measure it first
  The carried position is four scalars, so the COST per run does not grow with history. What is unmeasured is accuracy over a long chain (every rule in this corpus resolves inside four runs), and what does grow is the PROMPT, because the population is re-read whole -- the claim book is already 24 pct of this prompt's characters at 536 claims.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-23. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-23 — r001-warranty-dupe-watch. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with NO key configured reproduces the corpus byte-identically (python3 tools/build_corpus.py), passes all eleven pre-flight assertions (python3 -m evals.check_labels), writes the missed-run key and prices the cadence gap (python3 -m evals.missed_run), scores BOTH free floors (python3 -m evals.run --run-id b000-... --baseline, and again with --carry-floor), proves the wiring end to end (--stub) and serves the whole UI (python3 -m src.app). Six commands, plain python3, no install step -- requirements.txt names nothing because the kit imports nothing outside the standard library. ⚑ ON THIS KIT THE FREE PATH IS NOT A CONSOLATION PRIZE: it reproduces every published quality figure, because every published quality figure was measured without a provider.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per reading: one HTTP request carrying the assembled prompt, and the reply parsed to a level, a pattern, a count and a movement. Parsing the claim book, the campaigns and the dispositions, and advancing the carried position, all happen outside this clock and cost no network at all. Measured over all 160 readings of r001-warranty-dupe-watch at the published MAX_TOKENS = 24,000. ⚠︎ THE TAIL IS THE STORY AND IT IS NOT DRIVEN BY PAGE LENGTH: p95 (92,596 ms) is 3.5 times p50 (26,085 ms), while every reading is within a narrow band of every other in size. What drives a long reply is how much set arithmetic the run needs -- a dealer with two patterns and a declination to read is several times the work of a dealer with an empty book. ⚠︎ AND THESE FIGURES WERE MEASURED WHILE THREE OTHER KITS WERE RUNNING AGAINST THE SAME PROVIDER KEY, so they carry queueing that a solo run would not; they are an honest upper bound on this configuration rather than a clean measurement of it.

Current processWhat it replaces

Somebody working a warranty audit queue by hand every Monday: pulling each dealer's open claim book, working out which of the duplicates, repeat repairs, inflated labour lines and out-of-population recall claims were already raised weeks ago, reading the investigator's notes to see which findings were declined and whether the reason was a real one, and only then deciding whether this dealer is getting worse or quietly closing out.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• 4 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one dealers (4 weekly runs each) is

No split, and no chunking. The unit is a DEALER -- a run of four consecutive weekly readings processed strictly in order, because run 3's prompt contains a position produced by run 2. Each reading goes whole into one call, minus the one section no field asks for.

SetupWhat the setup figure measured

There is no index and no retrieval step. tools/build_corpus.py writes 160 readings and two answer keys from a fixed seed with no clock read and no model called; nothing is embedded, ranked or cached.

LicenceLicence

MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved; every dealer, principal, address, VIN, claim, operation code, recall campaign, investigator finding and note is invented here. Verified against the repository's own LICENSE file on 2026-08-23.

Bring your ownBring your own dealer readings

Replace src/programme.py FIRST, and not as an option. It is your programme's thresholds, level table and chargeback window, the shipped ones are invented, and data/gold.jsonl is literally its output -- change the rules and re-run tools/build_corpus.py and the answer key follows. Then point the corpus generator at your own dealers, or delete it and drop real readings into data/corpus/ named <DEALER>-W<n>.txt with the same seven section headings. src/segment.py, src/select.py, src/prompt.py and src/state.py read sections by name and do not care where the readings came from. Then set SCHEDULES in evals/run.py to the cadence you actually run, and evals/missed_run.py will price a missed one for nothing.

⚠︎ And what stops being true when you do: Every figure here is a property of a four-run corpus of 40 invented dealers, measured once. Three denominators are small: 24 prose-dependent readings, 13 INVESTIGATE readings, and 2 total errors in the best arm -- two rows is not a rate. ⚠︎ AND THE MARGIN THAT DECIDES WHETHER TO PAY FOR A MODEL IS ONE READING. 98.75 pct against 98.12 is 158 correct readings against 157, and both arms' errors sit in one storyline of nine. A book with a different mix of investigator wording moves that margin without either system having changed, and it could move it either way. The thing most likely not to survive contact with a real programme is the declination taxonomy -- the field every arm is worst on, and the one the catalogue row already records as unconfirmed.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 9,945 characters assembled, 3,757 are datas — 38% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Replayed from the kit's own src/prompt.py::build on DLR-0033-W2 with the exact carried position r001-warranty-dupe-watch recorded for that call ({"prev_level": "WATCH", "runs_at_review_or_above": 0, "prev_pattern": "DUPLICATE_CLAIM", "last_claim_seq": 100270}) -- the identical code path the run used, not a paraphrase. THE SPLIT IS IN CHARACTERS AND THE PAGE SAYS SO: per-part TOKEN counts were not measured, because measuring them means sending nested prefixes of the prompt to a provider and this kit spent no calls on it. tokens.input and tokens.output are the MEASURED averages over the scored run's 160 calls (409,625 in / 688,689 out). The stateless control's build of the same reading differs only inside the carried-position block, and evals/check_labels.py asserts every other byte is identical -- that is the entire experiment, and both arms of it are now fired.

Full promptThe prompt in full

You apply a written warranty audit programme to one dealer's claim book on one scheduled run. You answer with one JSON object and no other text.

You are reading one franchise dealer's warranty claim book on one scheduled run of an audit
programme, and deciding what the programme should report about that dealer THIS RUN.

The programme's rules, its thresholds and its chargeback window are reproduced in the reading below.
Apply them exactly as written. The rules depend on what the PREVIOUS run of this programme recorded
about this dealer, which you cannot see; what is known is stated under "Carried state" and is the
only history available to you. Do not assume anything about earlier runs beyond it.

How to decide:
 - Work out which claims in the Claim Book are COUNTABLE under each of the five patterns, using the
   thresholds in the rules.
 - Read the Investigator Disposition. A DECLINED finding removes the claims it names from the
   countable population only when the investigator's recorded reason is substantive -- the claims
   are genuinely separate work, correctly authorised, or already accepted by the manufacturer. A
   declination for an administrative reason -- the evidence was insufficient, nobody could be
   reached, the file was closed unworked -- leaves those claims countable. OPEN and QUEUED change
   nothing.
 - Of what is left, work out which claims are NEW this run: claim numbers above the number the
   carried state gives, plus any claim that crossed the parts clock between the previous run date
   and this one.
 - Take the pattern with the most new countable claims, apply the level table, then the consecutive
   runs rule, then decide the movement against the level last reported.

Answer with a single JSON object and nothing else:

{"level": "NONE|WATCH|REVIEW|INVESTIGATE",
 "pattern": "DUPLICATE_CLAIM|REPEAT_REPAIR|LABOUR_TIME_INFLATION|PARTS_NOT_RETURNED|OUT_OF_POPULATION_RECALL|NONE_DETECTED",
 "new_findings": <whole number of NEW countable claims attributed to that pattern this run>,
 "movement": "NEW|WORSENING|UNCHANGED|EASING|CLEARED",
 "rationale": "one sentence, naming the claims you counted and the rule you applied"}

"new_findings" counts only claims that were not already countable at the previous run; a claim that
was countable last run and is countable now has already been counted and is not new. "movement" is
this run's level compared with the level the carried state reports, and is CLEARED only when this
run's level is NONE and the previous one was not.

This programme surfaces a pattern for an investigator to work. Do not recommend a debit, a
chargeback or an audit -- none of those is yours or this programme's to make.

Carried state
------------------------------------------------------------------------------
At the previous run on 2026-07-06 this dealer was reported WATCH for DUPLICATE_CLAIM. It has not been at REVIEW or above on any run up to and including that one. The claim book then ended at CLM-100270; every claim numbered above that is new since the last reading.

Dealer reading
------------------------------------------------------------------------------
Synthetic Record
------------------------------------------------------------------------------
This file is synthetic. It was generated for an open evaluation kit and describes no real
dealer, person, vehicle, warranty claim, campaign or manufacturer. Dealer DLR-0033, run W2 (2026-07-13).

Dealer
------------------------------------------------------------------------------
Dealer code    : DLR-0033
Region         : South West
Franchise      : volume marque
Claim volume   : large (over 1500 claims a year)
Enrolled       : 2022-09-06
Run date       : 2026-07-13
Previous run   : 2026-07-06
Claims in book : 14 (every open claim paid in the last 90 days, re-read whole this run)

Audit Programme Rules
------------------------------------------------------------------------------
Countable claims. A claim in the book below is COUNTABLE against one of five patterns:
  DUPLICATE_CLAIM          two or more paid claims on the same VIN for the same failure code
                           within 30 days. The later claim of each pair is countable.
  REPEAT_REPAIR            3 or more paid claims on the same VIN for the same component
                           group within 180 days. The third and each later claim is countable.
  LABOUR_TIME_INFLATION    labour hours claimed above 1.5 times the published time.
  PARTS_NOT_RETURNED       a claim above 500.00 USD recording parts returned N, more than
                           21 days after payment. It becomes countable on the run that
                           crosses that clock, not on the run the claim first appeared.
  OUT_OF_POPULATION_RECALL a claim citing a campaign whose VIN sequence falls outside the
                           campaign's affected range, or whose build date falls outside the
                           campaign's build window. Both tests must pass for the claim to be in
                           population.

New this run. A countable claim is NEW when its claim number is above the number the previous
reading's book ended at -- stated under Carried state -- or when it crossed the parts clock since
the previous run date. Nothing else is new; a claim that was countable last run and is countable
now has already been counted.

Level. From the number of NEW countable claims attributed to the pattern:
  4 or more   INVESTIGATE
  2 or more   REVIEW
  1           WATCH
  0                 the level EASES one step from the level last reported
                    (INVESTIGATE -> REVIEW -> WATCH -> NONE).
A dealer at REVIEW or above for 3 consecutive runs including this one is INVESTIGATE
whatever the count.

Pattern. The pattern with the most new countable claims. On a tie, the earlier of the two in the
list above. When there are no new countable claims and the level is still above NONE, the pattern
last reported stands; at NONE the pattern is NONE_DETECTED.

Disposition (rule 6). A finding the investigator has DECLINED removes the claims it names from the
countable population ONLY when the recorded reason is SUBSTANTIVE -- the claims are genuinely
separate work, correctly authorised, or already accepted by the manufacturer. A declination for an
ADMINISTRATIVE reason -- the evidence was insufficient, nobody could be reached, the file was closed
unworked -- leaves the claims countable, because "we could not prove it" is not "it did not happen".
Findings marked OPEN or QUEUED change nothing.

Chargeback window. A claim cannot be charged back to the dealer more than 30 days after it was
paid. A finding raised after that date is a finding nobody can act on.

This programme surfaces patterns for an investigator. It debits nothing, charges back nothing and
opens no audit by itself.

Recall campaigns in force and their affected populations:
  RCL-2451  affected VIN sequence 010000-019999  built 2025-03-01 to 2025-08-31
  RCL-2478  affected VIN sequence 040000-044999  built 2025-09-01 to 2026-01-31

Claim Book
------------------------------------------------------------------------------
Every open claim on this dealer, as at the run date. This is the whole population, re-read;
it is not a list of what changed.

CLM-100098  VIN 2FADP82633A500272  op 64-1150  fail F22  comp BRK  labour 1.0h (pub 1.3h)  parts Y  built 2025-05-14  paid 2026-05-25  campaign -  340.29 USD
CLM-100133  VIN SHGCM70298C500276  op 51-2007  fail F58  comp ENG  labour 1.3h (pub 1.7h)  parts Y  built 2025-05-14  paid 2026-06-08  campaign -  187.80 USD
CLM-100201  VIN 2FADP36641D500277  op 64-1150  fail F58  comp BDY  labour 3.6h (pub 4.8h)  parts Y  built 2025-05-14  paid 2026-06-26  campaign -  1386.88 USD
CLM-100223  VIN 1WBAV36641D500279  op 64-1150  fail F80  comp BDY  labour 1.0h (pub 1.1h)  parts Y  built 2025-05-14  paid 2026-06-28  campaign -  1178.60 USD
CLM-100234  VIN 7HGCM41155B500278  op 64-1150  fail F11  comp HVA  labour 2.3h (pub 2.5h)  parts Y  built 2025-05-14  paid 2026-06-29  campaign -  1223.18 USD
CLM-100247  VIN 1WBAV36641D500279  op 42-1180  fail F80  comp BDY  labour 3.4h (pub 3.3h)  parts Y  built 2025-05-14  paid 2026-07-01  campaign -  994.99 USD
CLM-100270  VIN WWBAV36641D500273  op 42-1180  fail F58  comp HVA  labour 3.2h (pub 3.9h)  parts Y  built 2025-05-14  paid 2026-07-04  campaign -  325.14 USD
CLM-100288  VIN 8VWZZ70298C500280  op 13-0455  fail F17  comp HVA  labour 3.0h (pub 2.3h)  parts Y  built 2025-05-14  paid 2026-07-08  campaign -  1579.07 USD
CLM-100311  VIN 9WBAV70298C500281  op 33-4821  fail F44  comp BDY  labour 1.4h (pub 1.2h)  parts Y  built 2025-05-14  paid 2026-07-09  campaign -  1503.27 USD
CLM-100326  VIN 8VWZZ70298C500280  op 33-4821  fail F17  comp HVA  labour 1.1h (pub 1.4h)  parts Y  built 2025-05-14  paid 2026-07-10  campaign -  1480.38 USD
CLM-100327  VIN 5VWZZ36641D500282  op 27-3310  fail F17  comp SUS  labour 4.5h (pub 4.9h)  parts Y  built 2025-05-14  paid 2026-07-10  campaign -  2013.57 USD
CLM-100348  VIN WWBAV70298C500275  op 27-3310  fail F80  comp BDY  labour 1.1h (pub 1.0h)  parts Y  built 2025-05-14  paid 2026-07-11  campaign -  240.35 USD
CLM-100349  VIN 9WBAV70298C500281  op 27-3310  fail F44  comp BDY  labour 4.1h (pub 4.5h)  parts Y  built 2025-05-14  paid 2026-07-11  campaign -  848.34 USD
CLM-100363  VIN 5VWZZ36641D500282  op 27-3310  fail F17  comp SUS  labour 4.3h (pub 4.4h)  parts Y  built 2025-05-14  paid 2026-07-12  campaign -  916.70 USD

Investigator Disposition
------------------------------------------------------------------------------
Findings raised on this dealer since the programme last read its book, and what the
investigator did with them. A finding names the claims it was raised on.

FND-2006  raised 2026-07-09  DUPLICATE_CLAIM  status DECLINED
    claims  CLM-100326, CLM-100349
    note    "the second visit was a goodwill repair authorised separately; the two are correctly distinct claims"

Programme Notes
------------------------------------------------------------------------------
The programme extract ran clean; no records were rejected for this dealer.

Raw responseThe raw response

{"level": "WATCH", "pattern": "DUPLICATE_CLAIM", "new_findings": 1, "movement": "UNCHANGED", "rationale": "Counted CLM-100363 as the only new countable claim (later duplicate on VIN 5VWZZ36641D500282 for F17 within 30 days; CLM-100326 and CLM-100349 were removed by the substantive declination), so one new finding yields WATCH, unchanged from the prior WATCH."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch a dealer's warranty claims turning into a pattern — 160 dealer readings. Three tiers of one model family answered, and every answer was then graded Five different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

There is no LLM-as-judge in this kit and that is not a shortcut. Three of the four answered fields are closed sets -- four levels, six patterns, five movements -- and the fourth is a whole number of claims. A judge is for answers whose correctness is a matter of reading; these are matters of equality and set arithmetic, and asking a model to grade 'REVIEW == REVIEW' would add cost, variance and a second thing to be wrong. evals/scoring.py compares each cell exactly against a key src/programme.replay computed, and scores both free floors and the stub through the same function. The rationale field is not scored by anything, and this page says so where it publishes a reply.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that ran the 8 calibration calls and the 39 answered calls of the abandoned scored run is kept out of these tables per this estate's naming rule, so nothing here is what was paid. The real spend is recorded in the shared call ledger, not on this page.

Grading unitWhat the grading figure prices

Grading calls nothing. A measured $0.00, not an unpriced one -- and on this kit the same is true of every published QUALITY figure, because both floors are pure code. evals/missed_run.py prices a missed run for $0.00 as well.

The gradersFive ways to grade

READ THIS ROW BEFORE ANY MODEL FIGURE ON THIS PAGE, BECAUSE IT IS 0.63 POINTS BEHIND THE MODEL AND FREE. b001 is not a straw man and it is not a model in disguise: it is 167 lines of ordinary Python implementing the printed rules, given the same four scalars the model's prompt carries. It scores 98.12 pct on the level against the model's 98.75, ties it at 100.0 pct on the pattern, misses 0 of 57 raises against the model's 0, and raises 3 false alarms in 103 quiet readings against 2. ⚑ THE GAP BETWEEN THE TWO FLOORS IS THE ARGUMENT FOR CARRYING STATE, AND IT IS FOUR TIMES LARGER THAN THE GAP THE MODEL BUYS. b000 is the identical engine with the memory taken away: level 51.88 pct, 44 false raises, and a structural 0.0 pct on the counts that need memory. Handing a rule engine four scalars moves the level 46.24 points; handing the same problem to a language model moves it +0.63 more. ⚑ AND BOTH ARMS FAIL ON THE SAME CLAUSE. All 5 of b001's errors and both of the model's sit in the DECLINED_SUBSTANTIVE storyline, where a declination's reason has to be read as substantive or administrative. The keyword table lands on 12 of 16 wordings; the model reads 22 of the 24 prose-dependent readings against the floor's 21. That is the whole of what a model is buying here.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same dealer reading, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

⚑ THIS KIT DISCARDED CALLS AND PUBLISHES THE NUMBER, WHICH MOST DO NOT. The runs behind every figure on this page are 8 calibration + 160 scored + 160 stateless control + 40 missed-run arm. On top of those, 173 calls were BILLED and produced nothing publishable, reconciled against the shared ledger's 664 dispatches for this kit: 39 answered before the account emptied (preserved unaltered in docs/incomplete/, never scored), 22 from a launch that reported success and wrote no file, and 112 from its surviving twin racing the relaunch. A further 121 calls were REFUSED with 402 and billed nothing at all. Everything else -- both free floors, the wiring stub, the scorer, the pre-flight and the cadence measure -- is pure code and costs a measured $0.00. The dollar figures here are projections onto the shared Google Gemini 3 Flash card, not what anybody paid.

Cost driversWhat actually moves the bill

• PROVIDER-SIDE REASONING, 97.8 pct of output tokens on the scored run (673,517 of 688,689) and therefore most of the bill. It was left at the provider's default and this kit has not measured what turning it off does to the four fields.
• HOW MANY CLAIMS ARE IN THE BOOK, WHICH IS THE INPUT SIDE AND IS THE ONE THAT GROWS. This kit re-reads the whole open population every run, so the prompt scales with the dealer's 90-day book: input averaged 2560 tokens a reading across 160 readings, and the Claim Book section is 24 pct of the worked example's characters at 14 claims. A large dealer's book is several hundred, and unlike a per-document kit there is no chunking step to bound it.
• HOW HARD THE SET ARITHMETIC IS, ON THE OUTPUT SIDE. Output averaged 4304 tokens and the largest reply reached 18,053 -- 75 pct of the published ceiling -- on readings that differ little in length. The long ones are the runs carrying a declination to read.
• THE FIXED HEAD OF THE PROMPT, WHICH IS UNUSUALLY LARGE HERE BECAUSE THE RULEBOOK IS REPRINTED ON EVERY READING. The system message, the instruction and the programme rules are 5,923 of 10,148 characters -- 58 pct, byte-identical on all 160 readings. That is a deliberate transparency decision (the rules are invented, so they are printed to be disbelieved) and it is also the single biggest thing a prompt cache would discount.
• ⚑ AND THE LARGEST LEVER ON THIS PAGE IS STILL WHETHER YOU CALL A MODEL AT ALL. The free carry-aware floor scores 98.12 pct on the level for $0.00 against the model's 98.75. Every dollar here buys +0.63 points -- 1 reading of 160.

Your volumeWhat it costs at your volume

Linear in DEALER-RUNS, and the multiplier is the cadence rather than the corpus. One call per dealer per run: 40 dealers weekly is 2080 calls a year, and this run's 160 readings cost 2.2709 USD on the shared card, so ten times the set is about 22.71 USD. ⚠︎ A REAL DEALER NETWORK IS WHERE THIS ARITHMETIC STOPS BEING COMFORTABLE: at 0.014193 USD a reading, two thousand dealers watched weekly is about 28.39 USD a week -- 1,476.07 a year -- to buy +0.63 points of level accuracy over free code. The honest deployment is the model on the subset the floor cannot answer, which on this corpus is 24 readings of 160 and where the model is genuinely better (22 of 24 against 21). ⚠︎ AND THE INPUT SIDE GROWS, because the population is re-read whole: input averaged 2560 tokens on books of around 14 claims, and a large dealer's is several hundred. What does NOT scale is wall clock: runs within one dealer are strictly serial, and this run took 575 seconds of wall time for 160 calls at 12 workers.

Model choiceWhy this model

It is the tier the shared key is pointed at, and it is the cheap one -- the right place to start a question whose honest answer might be 'free code already does this'. On this corpus free code does 98.12 pct of it, and the cheap tier buys +0.63 points more. A more expensive tier was not run: the margin available above the free floor was 1.88 points before this run started, so there is very little room for a better model to demonstrate anything, and buying that room was not worth another 160 calls.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every model number on these pages: 160 readings, one completion call each, one tier. The 160-call stateless control, the 40-call missed-run arm, the 8-call calibration and roughly 173 discarded calls are accounted for separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

15 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pythe corpus generator — a swap seam

Generates 40 dealers x 4 weekly runs = 160 readings and 536 claims from a fixed seed (SEED = 20260823), across nine deliberately separated storylines. It writes each reading, reads it BACK through the runtime's own parser, and computes the answer key from that -- so the page and the key cannot disagree about what is on the page.

You change it to: Point it at your own dealers and claim extracts, or delete it and drop real readings into data/corpus/ named <DEALER>-W<n>.txt with the same seven section headings. Everything downstream reads by dealer and run and does not care where the readings came from.

# Generate the corpus and the answer key. Deterministic, no clock read, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
SEED = 20260823
RUN_DATES = ("2026-07-06", "2026-07-13", "2026-07-20", "2026-07-27")
RULE = "-" * 78
CAMPAIGNS = {
REGIONS = ("North", "Midlands", "South West", "South East", "Scotland", "Wales")

src/programme.pythe audit programme — a swap seam

The five countable-claim tests, the level table, the consecutive-runs rule and the declination clause, as pure code -- plus assess() and replay(), which ARE the answer key and which the runtime never calls. ⚠︎ Every threshold in it is INVENTED and the kit ships that as a stated blocker; the full rule text is reproduced on all 160 readings so it can be read, disbelieved and replaced.

You change it to: Every threshold, the level table, the consecutive-runs rule and the chargeback window are your programme's, not ours. Change them there and re-run tools/build_corpus.py, which recomputes the whole answer key from the same functions -- the key cannot drift from the rules because it is their output. ⚠︎ THIS IS THE FIRST FILE TO REPLACE, not an optional one: everything shipped is invented.

# The warranty audit programme, as pure code. The answer key, and the rule text the reading prints.

DUPLICATE_SPAN_DAYS = 30
REPEAT_SPAN_DAYS = 180
REPEAT_MIN_CLAIMS = 3
LABOUR_MULTIPLE = 1.5
PARTS_MIN_USD = 500.0
PARTS_CLOCK_DAYS = 21
CHARGEBACK_WINDOW_DAYS = 30
BOOK_WINDOW_DAYS = 90
LEVELS = ("NONE", "WATCH", "REVIEW", "INVESTIGATE")

src/state.pythe carried position — a swap seam

SEAM 2 -- the thing that makes this a monitor. Four scalars per dealer (the level last reported, consecutive runs at REVIEW or above, the pattern attributed, and the claim number the last book ended at), written from the arithmetic and never from the model's reply, and rendered into English for the prompt. It costs the same on run 40 as on run 3.

You change it to: load()/save() are a JSON file today. Point them at a table, a key-value store or the audit programme's own database and nothing else in the kit changes -- for_dealer() is the whole read interface and describe() is the only thing the prompt sees.

# The carried state -- the thing that makes this a monitor and not another classifier.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEFAULT_PATH = os.path.join(HERE, "data", "state.json")
def load(path=DEFAULT_PATH):
def save(store, path=DEFAULT_PATH):
def for_dealer(store, dealer_id):
def describe(state, prev_run_date=None):
def _ordinal(n):

src/segment.pythe section splitter

Splits a reading into its seven named sections. Pure code; evals/check_labels.py asserts all seven are present in all 160 documents before a run may spend.

# Split one dealer reading into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "Dealer", "Audit Programme Rules", "Claim Book",
RULE = "-" * 78
def split(text):
def body_of(text, name):
def render(secs):

src/select.pythe section selector — a swap seam

Decides which sections reach the model. Dealer Contact -- the principal's name, trading address, telephone and email -- is mapped by no field and is subtracted unconditionally by _fallback(), so the one section whose subject is a PERSON never leaves the machine even when a hint matches nothing.

You change it to: SECTION_HINTS maps a field to the sections it needs; NEVER_SENT names what is withheld whatever happens. ⚠︎ On a dealer network this is the load-bearing seam: check NEVER_SENT covers every section your readings carry that names a person, before anything spends.

# Pick which sections of a dealer reading are sent. Pure code -- the last deterministic step

BANNER = "Synthetic Record"
DEALER = "Dealer"
RULES = "Audit Programme Rules"
BOOK = "Claim Book"
DISPOSITION = "Investigator Disposition"
CONTACT = "Dealer Contact"
NOTES = "Programme Notes"
NEVER_SENT = (CONTACT,)
SECTION_HINTS = {

src/prompt.pythe prompt

Three parts: the fixed instruction, the carried-position sentence, and the selected sections in document order. The stateless build replaces the position block with a line saying no history is available -- byte-identical everywhere else, which is what makes the control a control.

# Assemble the one prompt this kit sends. No template engine, no framework -- string concatenation

INSTRUCTION = """\
def build(text, carried, prev_run_date=None, stateless=False):

src/adapters/__init__.pythe model adapter — a swap seam

SEAM 1 -- the model. Raw HTTP for every provider, standard library only. Returns input/output tokens, reasoning tokens and finish_reason beside the text, and separates a transport failure from an HTTP status so a dropped connection is retried rather than recorded as a lost reading. ⚠︎ It is also what refused this kit's scored run: a 402 Insufficient Balance is terminal, not transient, so it is raised rather than retried.

You change it to: .env plus the same run again. Any OpenAI-compatible provider or Anthropic.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/watch.pythe watch

One dealer, one run, one call. Parses the claim book, the campaigns and the dispositions off the page with regexes (the model is never asked to read a number), assembles the prompt, calls the model, parses the reply. Holds MAX_TOKENS = 24000, the published ceiling.

# One dealer, one scheduled run, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = ("You apply a written warranty audit programme to one dealer's claim book on one "
MAX_TOKENS = 24000
FIELDS = ("level", "pattern", "new_findings", "movement")
def documents():
def dealers():
def load_doc(doc_id):
CLAIM_RE = re.compile(

src/app.pythe local UI

One dealer, one run, its carried position and four columns, on 127.0.0.1:9001. Renders with no key. It shows the carried sentence verbatim and BOTH free floors beside the model's answer and the key, so a reader can see on which rows a model bought nothing.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
PORT = int(os.environ.get("PORT", "9001"))
GOLD_ROWS = {}
def carried_for(doc_id):
class H(BaseHTTPRequestHandler):
def main():

evals/baseline.pythe free one-reading floor

Everything a rule engine reading THIS week's book can compute, with no idea what last week said. 0 calls, $0.00, scored through the identical scorer.

# The two free floors. No model, no key, no network, and both scored through the same scorer.

ADMIN_MARKERS = ("insufficient", "out of time", "unable", "could not be reached",
SUBSTANTIVE_MARKERS = ("goodwill", "authorised", "approved", "campaign", "accepted",
def classify_note(note):
def _countable(book, run_date, campaigns, cleared):
def review(text, carried=None):

evals/baseline.pythe free carry-aware floor

The same rule engine given the same four scalars the model is given. It implements every printed rule except one -- whether a DECLINED note is substantive or administrative -- for which it uses an ordered keyword table. 0 calls, $0.00. ⚑ THIS IS THE HONEST COMPARATOR AND CURRENTLY THE BEST ARM THIS KIT HAS.

# The two free floors. No model, no key, no network, and both scored through the same scorer.

ADMIN_MARKERS = ("insufficient", "out of time", "unable", "could not be reached",
SUBSTANTIVE_MARKERS = ("goodwill", "authorised", "approved", "campaign", "accepted",
def classify_note(note):
def _countable(book, run_date, campaigns, cleared):
def review(text, carried=None):

evals/scoring.pythe scorer

Exact match per cell against the computed key, split ways an average would hide: the four fields, the raise directions counted apart, the memory-dependent subset, the prose-dependent subset, the first run against the later ones, and the recoverable exposure in dollars. No judge model.

# Score one arm against the computed answer key. Pure code -- no judge, no key, no network.

FIELDS = ("level", "pattern", "new_findings", "movement")
RAISED = ("REVIEW", "INVESTIGATE")
LEVEL_VALUES = set(PR.LEVELS)
PATTERN_VALUES = set(PR.PATTERNS) | {PR.NONE_DETECTED}
MOVEMENT_VALUES = set(PR.MOVEMENTS)
def _pct(n, d):
def _int_or_none(v):
def score(records, golds):

evals/missed_run.pythe cadence measure

Writes the SECOND answer key -- what the correct answers become when one weekly run never happens -- and prices the gap in claims and dollars. Free, no calls. ⚑ This is the half of a monitor that usually goes unmeasured.

# What a missed run costs. Free -- no model, no key, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
OUT_GOLD = os.path.join(HERE, "data", "gold-missed-w3.jsonl")
OUT_REPORT = os.path.join(HERE, "results", "cadence-missed-w3.json")
SCHEDULE = ("W1", "W2", "W4")
SKIPPED = "W3"
def chain_for(dealer_id, labels):
def main():

evals/check_labels.pythe pre-flight

Eleven things that must be true before a run may spend: the corpus shape, the seven sections, claim numbers in payment order, every declination note in exactly one pool, the key re-derived from the rendered pages, the memory-dependent flag verified in both directions, the two prompt builds byte-identical outside one block, the privacy guard red-proven in both directions, no money-moving code path, both floors answering every reading, and the missed-run key reproducing.

# Everything that must be true before a run may spend. Free -- no model, no key, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
GOLD_MISSED = os.path.join(HERE, "data", "gold-missed-w3.jsonl")
RUNS = ("W1", "W2", "W3", "W4")
EXPECT_DEALERS = 40
EXPECT_DOCS = 160
FORBIDDEN_CALLS = ("debit(", "charge_back(", "raise_chargeback(", "post_chargeback(",
FORBIDDEN_ON_PAGE = ("last reported", "previously reported", "previous level", "prev_level",
class Refused(SystemExit):

evals/run.pythe run harness — a swap seam

40 dealer chains, four strictly-ordered runs each, 12 concurrent workers. The position advances even for a run whose CALL failed, so one transport error cannot turn into four scored failures -- which is exactly the branch the 402 exercised.

You change it to: SCHEDULES maps a name to the sequence of runs the programme actually performs. 'weekly' and 'missed-w3' ship; add your own and evals/missed_run.py prices it against the weekly one for nothing. ⚠︎ This is the seam most kits in this series do not have, and it is where a monitor's real behaviour lives.

# Run the watch over the 40 dealers and write one result file. THIS SPENDS MONEY.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
GOLD_MISSED = os.path.join(HERE, "data", "gold-missed-w3.jsonl")
SCHEDULES = {
def load_gold(path):
def stub_complete(cfg, system, user, max_tokens=1024):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2560 input and 4304 output tokens per dealer reading, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

No untrusted external input surface in the shipped version. Every field that reaches the prompt comes from tools/build_corpus.py's fixed seed, including the Programme Notes, which are one of five fixed sentences. src/select.py names the notes as the field an outside party WOULD influence in a real deployment -- prose an analyst types into a review queue -- and sends them deliberately, so the surface is visible rather than hidden. ⚠︎ ON THIS VERTICAL THAT SURFACE IS UNUSUALLY ATTRACTIVE: a dealer under investigation has a direct interest in a note that talks the level down, and rule 6 means one sentence can legitimately remove claims from the countable population. On this corpus nobody outside the repository wrote a byte of it.

The provider key lives only in .env (gitignored) or the shared repo-root .env, read by src/config.py and never logged. config.sources() reports WHICH file a value came from and never the value. config.save() opens the file 0600 before the write rather than chmod-ing it after, so the key never exists at the default umask even for an instant. src/app.py redacts the key and the base URL out of any provider error before it reaches the browser -- which mattered here: the error the page actually renders today is a 402 body from the provider.

The experimentWe did not attack it -- and the boundary that matters most on this vertical was proven by breaking it on purpose

An indirect prompt injection needs a field an outside party controls that reaches the prompt, and this corpus has none. So the five rows above are boundaries, not payloads. One of them was red-proven: src/select._fallback was replaced with the naive or list(secs) every sibling kit once shipped, the condition that actually reaches the fallback was reproduced -- a reading with no Synthetic Record banner and no Programme Notes, which is what a raw dealer-management-system extract looks like -- and all 160 readings leaked the principal's name, trading address, telephone and email. Restoring the guard took it back to 0 of 160. ⚠︎ THE REPRODUCTION IS THE HALF THAT MATTERS. Swapping the guard alone changes nothing on today's corpus, because every hint names a section every reading carries, so the fallback is unreachable and the red-proof would have measured ZERO and reported a green. A guard that has never been MADE to fail is a guard nobody has tested. Confirmed by assertion and by reading the recorded runs, not by an attack trial, on 2026-08-23 -- no attack run was fired; see redteam.why.

Each boundary above was checked by running an assertion or by reading a recorded run, not by an attack trial -- there is no untrusted field on this corpus to construct a payload against. One of the five is red-proven, meaning the guard was removed and the failure observed rather than merely asserted while passing, and its red-proof reproduces the condition that reaches the guard rather than relying on today's corpus to reach it.

The result0 attack trials, five boundaries checked -- and the privacy boundary red-proven by removing the guard, reproducing the condition that reaches it, and watching all 160 readings leak a dealer principal's name, trading address, telephone and email.

The Programme Notes ARE the field an outside party would influence in a real deployment, and this kit sends them rather than hiding them -- but on this corpus they are one of five sentences chosen by a seeded generator, so there is nothing adversarial in them to catch. ⚠︎ A version pointed at real analyst prose reopens the question sharply, because rule 6 gives a sentence the power to remove claims from the count. What IS measured is the privacy boundary, in both directions, before every run.

HonestyWhat this does not prove

• Whether a real programme's analyst notes -- prose somebody writes freely into a review queue -- would carry an instruction the model follows. Not applicable to the shipped corpus, and the first thing to attack if this is pointed at a real book. It is a sharper question here than on most kits because rule 6 lets one sentence remove claims from the countable population.
• Whether the no-propagation property holds under a wrong answer MID-CHAIN. Both of the scored run's errors are on run 4, the last of their chains, so nothing downstream of them exists to be corrupted. The stateless control errs mid-chain and does not propagate, which is consistent with the property and is not a demonstration of it.
• Whether the state file is safe under concurrency. src/state.save() replaces atomically, which is correct for one writer and is not a concurrency model; two schedulers advancing one dealer have never been run and would race. A lost run does not reset the position, it fails to advance it -- so the next run reads a stale claim number and re-raises claims already counted.
• Whether the grep-shaped no-money-movement assertion would catch a path written under a name it does not know. It asserts the absence of ten call names; a path called something else would pass it.
• Whether a truncated reply can ever be partially trusted. No run has hit the published ceiling -- the largest reply reached 75 pct of it -- so src/watch._parse's outright rejection of an unparseable reply has never been exercised on a real truncation.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

No debit, no chargeback, no audit opened, non-configurable. This kit produces a level, a pattern, a count of what is new and a movement, for an investigator to work. It never debits a dealer, raises or posts a chargeback, claws back a claim, adjusts a claim value or opens a case, and there is no setting that makes it. Separately: the carried position is written by code from the parsed claim book and the previous position; the model's four answers are scored and are never written back into the history the next run is judged against.

Everywhere and nowhere -- it is a property of what is ABSENT. The only writers in the kit are evals/run.py (results/*.json), evals/missed_run.py (a second answer key and a cadence report) and src/state.save (data/state.json). evals/run.py -> src/programme.assess is the only thing that touches the position, and it is called after every run including one whose call FAILED.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• IT IS NOT A CHECK ON WHETHER THE ANSWER IS RIGHT. The position advances correctly whatever any reader says, which means a wrong level is reported wrongly for that run and nothing catches it -- see Eval.taxonomy's five declination misses, which this guardrail does nothing to prevent.
• IT IS NOT AN APPROVAL GATE, BECAUSE THERE IS NOTHING TO APPROVE. There is no apply path to gate: nothing here moves money, so there is no 'undo' because there is nothing that could 'do'. A deployment that adds one adds the gate at the same time or it has no guardrail at all -- and on this vertical the thing being gated is a debit against a named business.
• ⚠︎ IT IS NOT A GUARANTEE ABOUT THE PROGRAMME ITSELF. Every threshold, the level table, the consecutive-runs rule and the chargeback window in src/programme.py are invented. A watch that is arithmetically perfect against an invented programme is arithmetically perfect against nothing.
• IT IS NOT A GUARANTEE THAT THE QUEUE IS FAIR TO THE DEALER. The output names a business and asserts a pattern. The free one-reading floor raises 44 false alarms in 103 quiet readings and the carry-aware one raises 3; nothing in this kit tells a dealer they were raised, and nothing measures what a wrongly-raised dealer experiences.
• It does not make the state file safe to share. One writer, one file, replaced atomically -- two schedulers on one dealer would race, and a lost write means the next run re-raises claims already counted.
• It does not validate the declination taxonomy, which is the field every arm is worst on and the one the catalogue row already records as unconfirmed.

WatchedWhat is watched, and why that one

The kit measures 56 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 49 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (gate_recall 1.0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

7 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

triage · with the model in the path — 6 runs. Columns here are only ever compared with each other.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 7 chips that all say so.

DeviationsWhat deviated

0 breaches across 7 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

⚑ THIS KIT HAS TWO CADENCES AND THEY ARE DIFFERENT THINGS. THE PROGRAMME'S: weekly, and evals/missed_run.py measures why -- a countable claim has 9 days of headroom against the chargeback window, a weekly run is inside it and a fortnightly one is not, and one missed run costs 6 claims worth 9,779.29 USD permanently. Re-derive it for your own constants; it is free and takes a second. THE EVAL'S: re-run evals/check_labels.py (free) on any change to tools/build_corpus.py, src/programme.py, src/segment.py or src/select.py -- it refuses to let a run spend if the corpus, either answer key, the experimental control, the privacy guard or the no-money-movement property has moved. Re-run both free floors together (free) on any change to evals/baseline.py: the headline is a DIFFERENCE, so one floor re-run alone is not comparable with the other's old figure. Re-run evals/missed_run.py (free) whenever a threshold or a schedule changes. Re-run the scored eval and its stateless control together (paid, 320 calls) on any change to src/prompt.py or src/state.py.

In one lineNo framework, deliberately

No framework and no dependencies at all -- json, os, re, random, datetime, argparse, urllib, http.server and concurrent.futures from the standard library. requirements.txt names nothing and says in a comment that the emptiness is load-bearing: a line there means something under src/ or evals/ imports it.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph, and the one place it looks like one is a straight line. Each dealer is a chain of four steps -- run 1, 2, 3, 4 -- with no branching and exactly one edge between consecutive runs, carrying four scalars. Different dealers never touch. A framework would add an orchestrator to a for-loop that already runs 40 chains wide. ⚑ AND THE EDGE IS CHEAP TO REBUILD, which is the unusual part: because the position is arithmetic rather than model output, --runs W4 reproduces the whole chain for free and spends only on the step being measured.

The other sideWhat a framework costs you

• No scheduler. This is a monitor that measures a cadence and does not implement one -- it is invoked, not woken. ⚠︎ On this kit that gap is priced rather than hand-waved: 6 claims worth 9,779.29 USD per missed run.
• No state store worth the name. data/state.json is one file replaced atomically; a framework would bring a checkpointer with a concurrency model, which this does not have.
• No observability beyond what evals/run.py prints and writes to results/. There is no tracing and no dashboard integration.
• No retry/backoff beyond src/adapters' own bounded retry, which covers a busy provider and a dropped connection and nothing else -- and deliberately does NOT cover a 402, which is terminal.
• No budget guard beyond src/budget.py's daily call cap, which counts calls, not dollars, and can see neither an account balance nor a second copy of the same run. ⚠︎ THAT IS THE GAP THIS KIT PAID FOR TWICE: the balance ran out mid-run (39 calls billed for nothing), and then a launch that appeared dead was not, so two copies of one run raced each other and double-billed until one was killed. The second hazard is named in budget.py's own docstring and not solved there.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-warranty-dupe-watch on the same tier, memory removed (THE CONTROL), 2026-08-23. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 5 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the population is re-read whole on each scheduled run, and the prior run's carried state is what makes a reading a CHANGE.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved; every dealer, principal, address, VIN, claim, operation code, recall campaign, investigator finding and note is invented here. Verified against the repository's own LICENSE file on 2026-08-23. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe level, the pattern, the new-finding count and the movement, per reading, exact match against the computed answer key

For each of the 160 readings and each of the four answered fields, did the reply equal the computed key? Level, pattern and movement are compared exactly; the count is compared as a whole number, and a reply that is not a whole number is a MISS rather than a zero -- 0 is a meaningful answer here (it is what an easing dealer gets) and a parse failure must never be scored as one.

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model. The same function both free floors and the wiring stub are scored through.

Every grader on these pages scored the same 480 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 160 per field. A reading whose reply did not parse counts as a MISS in every field, never as an exclusion.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, computed by src/programme.replay over the RENDERED pages at generation time. This grader IS the reference, so its own TPR and TNR are not measured -- scoring it against itself would be circular.

How tight can the band be? No tuned threshold anywhere. The denominators are printed beside every rate because three of them are small: 24 prose-dependent readings, 13 INVESTIGATE readings, and 5 total errors in the best arm.

Cadence: Re-run evals/check_labels.py (free) on any change to the corpus generator, src/programme.py, src/segment.py or src/select.py. Re-run both free floors (free) on any change to evals/baseline.py. Re-run the scored eval (paid, one call per reading) on any change to src/prompt.py or src/state.py -- both change what the model is told. Re-run evals/missed_run.py (free) whenever a threshold or the schedule moves, because the cadence answer is arithmetic over both.

The decisionWhen to reach for it

In one lineThe two directions, counted apart: dealers that earned a raise and were reported quiet, and quiet dealers put on the queue

Of the 57 readings whose correct level is REVIEW or INVESTIGATE, how many were reported at REVIEW or above? And of the 103 quiet readings, how many were put on the queue anyway? 103 of 160 readings are quiet, so an arm reporting NONE to everything would score 64.4 pct on any per-reading accuracy while surfacing nothing.

MethodHow the test was run

evals/scoring.py::score, the same pass as the reference grader -- a slice of the same cells, not a second comparison.

Every grader on these pages scored the same 480 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

detection = raised-correct / readings that earned a raise. false-raise rate = quiet readings reported at REVIEW or above / quiet readings. Never averaged.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the same data/gold.jsonl, sliced by whether the gold level is REVIEW or above.

How tight can the band be? 57 raised readings and 103 quiet ones, so one row is 1.75 and 0.97 points respectively. Nothing is tuned.

Cadence: Whenever the reference grader runs -- it is the same pass.

The decisionWhen to reach for it

In one lineThe readings whose answer cannot be derived from their own page

On the 93 readings where the same page with no carried position produces a different answer, was the level right and was the count right? This is the subset the whole monitor argument rests on. ⚑ IT IS A MEASUREMENT, NOT A LABEL: tools/build_corpus.py flags it by computing the answer twice, and evals/check_labels.py verifies the flag in BOTH directions, so the subset can be neither padded nor quietly trimmed.

MethodHow the test was run

evals/scoring.py::score, the same pass as the reference grader.

Every grader on these pages scored the same 480 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

hits among cells where gold.memory_dependent is true, divided by 93, reported separately for the level and the count.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the same data/gold.jsonl, restricted to cells the generator computed as memory-dependent and check_labels re-verified in both directions.

How tight can the band be? 93 readings. One row is 1.08 points.

Cadence: Whenever the reference grader runs -- it is the same pass.

The decisionWhen to reach for it

In one lineThe readings where an investigator's sentence decides the answer

On the 24 readings carrying a DECLINED finding, was the level right? Rule 6 says a declination removes its claims from the countable population only when the reason is substantive, and the reason is a sentence. ⚑ THIS IS THE ONLY GRADER ON WHICH A LANGUAGE MODEL HAS ANYTHING TO SELL IN THIS KIT.

MethodHow the test was run

evals/scoring.py::score, the same pass as the reference grader.

Every grader on these pages scored the same 480 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

hits among cells where gold.prose_dependent is true, divided by 24.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the same data/gold.jsonl, restricted to the 24 readings whose Investigator Disposition carries a DECLINED finding.

How tight can the band be? 24 readings. One row is 4.17 points, so nothing finer than about four points means anything here.

Cadence: Whenever the reference grader runs -- it is the same pass.

The decisionWhen to reach for it

In one lineWhat one missed run costs, in claims and in dollars

For every claim the weekly schedule counts as new somewhere, was the chargeback window still open on the run that first counted it -- and is it still open when one weekly run never happens? ⚑ THIS IS THE HALF OF A MONITOR THAT USUALLY GOES UNMEASURED. Four sibling kits in this estate carry state and none of them runs on a clock.

MethodHow the test was run

evals/missed_run.py, in-process, free. It also writes data/gold-missed-w3.jsonl -- a SECOND answer key, because the correct answers under a fortnightly gap are different answers and not worse ones.

Every grader on these pages scored the same 480 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

a claim is recoverable when the run that first counted it falls within 30 days of the claim's payment date. Compared between the weekly schedule and a schedule with run 3 removed. No model on either side.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl and data/gold-missed-w3.jsonl, both computed by src/programme.replay and both re-derived by evals/check_labels.py.

How tight can the band be? 6 claims lost is a count, not a rate. It is entirely determined by which claims cross the parts clock in the skipped week, which is a property of the generator.

Cadence: Re-run whenever a threshold in src/programme.py or a schedule in evals/run.py changes. Free.

The decisionWhen to reach for it