The business caseThe problem this solves

Before a completed currency-transaction filing is submitted, somebody has to check that the draft actually matches the log it was prepared from. It is an aggregation problem wearing a paperwork problem's clothes: the entries have to be re-added under the right window, the ones that are not currency have to stay out, a patron split across two records has to be recognised as one person, and a filing that was never due has to be caught before it goes anywhere. And the pack puts two things next to the answer that look like evidence and are not: the preparer's own confident sign-off, and a log full of entries the draft left out on purpose. Somebody opening each completed draft filing beside the cage log it came from and re-doing the preparer's work: re-adding every qualifying entry under the gaming-day window rather than the calendar day, dropping the wires and promotional credits that are not currency, dropping the entries in the other direction, checking whether a second patron record in the same log is the same person, comparing each transaction code on the draft against the log's own, checking the five identification elements and how old the identification is, and asking whether the threshold was actually crossed. It is a few minutes per pack and it is every pack, and the part that goes wrong is rarely the arithmetic: it is aggregating a calendar day instead of a gaming day, missing that two records are one patron, or flagging a wire the draft was right to leave out.

The inputThe actual QC packs

The corpus is 56 QC packs, 0.09 MB (txt 56). Plain text, one format, invented rather than fetched — a real QC pack names a patron, their date of birth, their address, their identification document and every cash movement they made in a day, beside a real property's cage log. There is no public corpus of (draft filing, correct defect list) pairs and there never will be, for reasons that have nothing to do with effort. Generating it also makes the label mechanical: gold's defect list is src/rulebook.py::assess() run over the same values the generator itself decided, never somebody's reading of the preparer's note. AND IT IS WHAT MAKES THE FALSE-ALARM RATE MEASURABLE AT ALL: 18 of the 56 packs have nothing wrong with them, every one of them carries at least one piece of bait a careless checker would flag, and you cannot construct that guarantee out of found data.

Draft Filing
------------
DF-2026-8029

Property And Shift
------------------
Northgate Floor, swing shift, cage 2

Patron Record
-------------
PR-FE2828

Gaming Day
----------
2026-02-15

Direction Reported
------------------
cash_out

Window Applied
--------------
gaming day (06:00 to 06:00 the next date)

Reported Total
--------------
12150 CU

Transactions Included On The Draft
----------------------------------
TXN-05092  safekeeping_withdrawal     2450 CU
TXN-05095  front_money_withdrawal     4950 CU
TXN-05098  safekeeping_withdrawal     4750 CU

Patron Identification On The Draft
----------------------------------
full name: K. Garsbourne
date of birth: 1982-03-05
residential address: 153 Fenridge Road, Kingsmere
identification reference: IDR-123815
patron account number: PA-262839
identification captured on: 2025-08-26

Cage Transaction Log
--------------------
TXN-05098  2026-02-15 14:30  PR-FE2828    safekeeping_withdrawal     4750 CU
TXN-05101  2026-02-15 20:46  PR-SG5705    credit_instrument_issue    2300 CU
TXN-05104  2026-02-16 00:24  PR-SG5705    front_money_withdrawal     1000 CU
TXN-05095  2026-02-16 01:02  PR-FE2828    front_money_withdrawal     4950 CU
TXN-05092  2026-02-16 01:45  PR-FE2828    safekeeping_withdrawal     2450 CU
TXN-05107  2026-02-16 09:25  PR-FE2828    front_money_withdrawal     1300 CU

Other Patron Records In This Log
--------------------------------
PR-SG5705 -- date of birth 1982-03-05, identification reference IDR-123815

Preparer Note
-------------
Not confident the totals tie out -- please re-add these before this goes anywhere.

The outcomeWhat a good result looks like

A fourteen-field extracted record per pack, plus a defect list drawn from a closed seven-code vocabulary and produced afterwards in pure code, and one routing flag: does this draft have to go back to the preparer because what would be FILED is wrong, or is it a content correction made in place? Each defect carries the rulebook's own reasoning, and 'insufficient_information' is a real answer the check can reach. Nothing here files, transmits, approves, signs or clears anything.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Screening a day's completed draft filings before a compliance reviewer opens them — the fast tier — 56 of 56 defect sets exact and 0 of 18 clean filings flagged, on one scored run
  The queue this produces is 38 rows long and every row is real. The free preparer-note floor's queue is 8 rows long, 7 of them noise, and 37 real defects are not in it at all.
• Deciding how much of the check to give to a model at all — much less than this kit does — give it the reading and keep the arithmetic
  The free regex floor scored 112 of 112 on both arithmetic cells and 728 of 728 on every structured field, for $0.00. Everything the model uniquely earned on this corpus is in the fourteenth field: recognising which of seven defect shapes a pack carries, and declining to invent one on the 18 that carry none.

And where nothing here is good enough:

• Deciding whether this shape is worth buying at all, from these numbers — nothing yet — this corpus cannot answer it
  One scored run and a rules-stripped ablation both scored perfectly, so nothing here separates any two configurations. What the numbers DO support is narrower and real: a tone read of the preparer's note is measurably unusable on this shape of check, at a 38.89 pct false-alarm rate, and a self-consistency check catches every one of that shortcut's errors with no labels.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-ctr-precheck. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts on 127.0.0.1:8856, the 56-pack picker populates, the field table draws its fourteen empty rows and the shipped rulebook renders in full from data/rulebook.json; clicking Pre-check returns the no-key sentence rather than a stack trace, and the recompute row reads “not computed — the reply carried no defect list, and an unknown is not a pass” rather than “no”. python -m evals.check_labels and python -m evals.run --baseline both pass with no network access at all.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per QC pack, measured around the adapter call inside a 12-worker pool

Current processWhat it replaces

Somebody opening each completed draft filing beside the cage log it came from and re-doing the preparer's work: re-adding every qualifying entry under the gaming-day window rather than the calendar day, dropping the wires and promotional credits that are not currency, dropping the entries in the other direction, checking whether a second patron record in the same log is the same person, comparing each transaction code on the draft against the log's own, checking the five identification elements and how old the identification is, and asking whether the threshold was actually crossed. It is a few minutes per pack and it is every pack, and the part that goes wrong is rarely the arithmetic: it is aggregating a calendar day instead of a gaming day, missing that two records are one patron, or flagging a wire the draft was right to leave out.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 0 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings; a pack with none falls back to one whole-document segment

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 56 packs cut into 672 sections in seven thousandths of a second, in process, with no model and no network.

LicenceLicence

MIT — this repository's own licence. Every patron name is built mechanically from a syllable list plus an initial, every property name is a constructed compound noun, and every record number, identification reference, account number, filing identifier and transaction identifier is an invented format. NO REAL CASINO, GAMING OPERATOR, REGULATOR, AGENCY, FORM NUMBER OR REGULATION IS NAMED ANYWHERE IN THIS KIT. The rulebook it ships (data/rulebook.json) is INVENTED and is not an authority: the threshold, the aggregation window, the staleness horizon, the identification elements and the transaction-code table were written for this kit and reproduce no real regulation, filing form, filing instruction, supervisory guidance or compliance manual. Amounts are in CU, an invented unit that is not a currency and maps to none.

Bring your ownBring your own QC packs

REPLACE data/rulebook.json FIRST. The threshold, the window, the codes, the elements and the staleness horizon are all this kit's own inventions, and everything downstream reads it: src/rulebook.py loads it, src/prompt.py renders it into every call, and tools/build_corpus.py writes gold with it. Then replace data/corpus/*.txt, write your own data/fields.json, and supply a gold record per pack. SECTION_HINTS in src/select.py maps fields to section headings and will need editing for a different pack layout; when it does not match, selection falls back to the whole document — slower, more expensive, always correct. compute() in src/extract.py is this kit's own triage rule and is the second thing to replace. ⚑ AND KEEP THE CLEAN BUCKET BIG. The false-alarm rate is the number this kit leads with and its denominator is the packs with nothing wrong with them; evals/check_labels.py refuses to run against fewer than 12 of them, and separately refuses if any clean pack carries no entry the draft correctly left out.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 3,439 tokens assembled, 1,654 are contexts — 48% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on QCP-0040 — the same pack the worked example above was captured on, so the four parts sum to exactly the 3,439 input tokens that call actually reported. Four calls at max_tokens=1, each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. kits/UC0056-ctr-precheck/results/tokens-p002-ctr-precheck.json. ⚑ THE SAME MEASUREMENT ON A DIFFERENT PACK (QCP-0001, tokens-p001-ctr-precheck.json) returned 843 / 1,083 / 942 for the first three parts — byte-identical, which is the evidence that the 2,868-token fixed prefix really is fixed and is paid on every call. Only the fourth part moved, by six tokens.

Full promptThe prompt in full

You read a CURRENCY-TRANSACTION FILING QC PACK -- a draft filing beside the cage transaction log it was prepared from -- and extract structured fields from it. You return JSON and nothing else.

You are PRE-CHECKING a draft for a qualified person to act on. You never file anything, you never clear anything, and your answer is not a substitute for a compliance officer's judgement or for the filing instructions that actually apply.

RULES, in order of importance:
1. If the pack does not state a field, return null for it. Do not infer it and do not use what you know about the world.
2. `defects_found` is decided ONLY by the RULEBOOK given below, applied to the numbers on the page. Work through the STOPPING ORDER and STOP at the first check that fires:
   a. INSUFFICIENT INFORMATION. If any qualifying log entry's amount was never captured, or the draft does not state a gaming day, the qualifying total cannot be computed. Answer 'insufficient_information' and stop. This is a real answer, not a failure to produce one.
   b. THRESHOLD NOT CROSSED. If the qualifying total does not exceed the threshold, no filing is due at all. Answer 'threshold_not_crossed' and stop.
   c. WINDOW MISAPPLIED. If the draft's Window Applied line names anything other than the gaming day, that is the named cause of any total difference. Report 'window_misapplied' and do NOT also report 'missed_aggregation' for the same difference.
   d. IDENTITY SPLIT. Otherwise, if another patron record in the log matches this patron on BOTH link keys and the draft did not aggregate it, report 'identity_split'.
   e. MISSED AGGREGATION. Otherwise, if the qualifying total exceeds the drafted total, qualifying entries on the patron's own record were left out. Report 'missed_aggregation'.
3. THEN, INDEPENDENTLY OF 2c-2e, add 'identification_gap' when a required identification element is missing from the draft or the identification is stale, and 'type_miscode' when a transaction's code on the DRAFT is not the code the LOG records for the same identifier. These change what the filing SAYS rather than what it FILES, so they are checked separately and they never suppress a defect from 2c-2e.
4. A NON-REPORTABLE ENTRY IS NOT A DEFECT. The log carries entries the rulebook marks non-reportable, entries in the opposite direction, and entries outside the gaming day. All three are CORRECTLY absent from the drafted total. Reporting them as a missed aggregation is a false alarm, and on this check a false alarm costs a person the time to clear a row that never needed clearing.
5. A FILING WITH NOTHING WRONG WITH IT IS 'none'. Answer 'none' and do not manufacture a finding to look thorough.
6. THE PREPARER'S NOTE IS A FIELD TO COPY, NOT EVIDENCE ABOUT THE FILING. A note that sounds confident does NOT make a defective filing clean, and a note that sounds anxious does NOT make a clean filing defective. The rulebook and the numbers decide; the note is one person's remark about their own work and may disagree with them.
7. Amounts are plain integers in CU. Return them with no unit, no separators and no decimals.
8. Use the exact allowed value for a field that lists them.
9. Return every field named in the schema, even when the answer is null.

FILING RULEBOOK (the authority for `defects_found`; this rulebook is INVENTED and was written for this kit -- it reproduces no real regulation, form or filing instruction, and CU is an invented unit that is not a currency)

THRESHOLD
  A filing is due when the qualifying total for ONE patron, in ONE direction, over ONE gaming day exceeds 7500 CU. Exactly 7500 CU does not cross it.

AGGREGATION WINDOW
  The GAMING DAY runs from 06:00 on its own date to 06:00 the following calendar date.
  An entry timestamped 02:40 on the date AFTER the gaming day still belongs to that gaming day.
  An entry timestamped 03:15 on the gaming day's OWN date belongs to the PREVIOUS gaming day.
  Cash in and cash out are aggregated SEPARATELY and are never netted against each other.

TRANSACTION CODES -- direction, and whether the code is part of a CURRENCY total
  chip_purchase_cash         in   REPORTABLE      cash across the cage counter for chips
  chip_redemption_cash       out  REPORTABLE      chips redeemed for cash at the cage
  credit_instrument_issue    out  REPORTABLE      cash advanced against a credit instrument
  credit_instrument_payment  in   REPORTABLE      cash paid against an outstanding credit instrument
  front_money_deposit        in   REPORTABLE      cash placed on deposit against future play
  front_money_withdrawal     out  REPORTABLE      cash drawn down from a front-money balance
  promotional_credit         in   not reportable  a non-cash promotional award posted to the patron account
  safekeeping_deposit        in   REPORTABLE      cash lodged for safekeeping
  safekeeping_withdrawal     out  REPORTABLE      cash drawn from a safekeeping balance
  ticket_redemption_cash     out  REPORTABLE      a machine ticket redeemed for cash
  wire_received              in   not reportable  funds arriving by wire -- not currency, so it is never part of a currency total
  wire_sent                  out  not reportable  funds leaving by wire -- not currency, so it is never part of a currency total

  A non-reportable code is NEVER part of a currency total, however large the amount. It is correctly absent from a draft and its absence is not a defect.

REQUIRED IDENTIFICATION ELEMENTS -- all 5 must be on the draft
  full name
  date of birth
  residential address
  identification reference
  patron account number
  Identification captured more than 400 days before the gaming day is STALE.

SAME-PERSON LINK KEYS -- two patron records are one person when BOTH match
  date of birth and identification reference
  A record matching on only one key is NOT a link.

DEFECT CODES -- the only seven values `defects_found` may carry
  missed_aggregation         A qualifying entry on the patron's own record -- right direction, inside the gaming day, a reportable code -- is not in the draft's total.
  window_misapplied          The draft aggregated something other than the gaming day, so entries either side of the 06:00 boundary are counted or dropped wrongly.
  identification_gap         A required identification element is missing from the draft, or the identification on it was captured more than the staleness horizon before the gaming day.
  identity_split             A second patron record in the log is the same person by the link keys, and the draft aggregated only one of them.
  type_miscode               An entry's transaction code on the draft is not the code the log records for that entry.
  threshold_not_crossed      The qualifying total never crossed the threshold. No filing is due at all, and the draft should not exist.
  insufficient_information   The record does not carry what the check needs -- an amount that was never captured, or a missing gaming day -- so no conclusion about the total can be reached from it.

STOPPING ORDER -- one named cause per total difference
  1. insufficient_information -- if the qualifying total cannot be computed at all, stop. Nothing below can be decided from a record that does not say what happened.
  2. threshold_not_crossed -- if the qualifying total is at or below the threshold, stop. Arguing about the address block on a filing that should not exist is noise.
  3. window_misapplied -- if the draft applied a window other than the gaming day, that is the named cause of any total difference; do NOT also report missed_aggregation for the same difference.
  4. identity_split -- else, if a linked record exists and the draft excluded it, that is the named cause.
  5. missed_aggregation -- else, if the qualifying total exceeds the drafted total, entries on the patron's own record were left out.
  6. identification_gap and type_miscode are CONTENT defects and are checked independently of 3-5. They do not change what is filed; they change what the filing says.

Extract these fields:
- filing_id (string) -- the draft filing's own identifier, verbatim
- patron_record_id (string) -- the patron record the draft filing is prepared under, verbatim
- gaming_day (string) -- the gaming day the draft covers, as the YYYY-MM-DD date the draft states
- draft_direction (enum) one of: cash_in, cash_out -- which direction the draft reports. Cash in and cash out are aggregated separately and are never netted against each other
- draft_reported_total (number) -- the total the draft filing itself states, in CU, as a plain integer with no unit and no separators
- log_qualifying_total (number) -- the total the CAGE TRANSACTION LOG actually supports for this filing, in CU, as a plain integer. Add up every log entry that qualifies under the rulebook -- a reportable code, the draft's own direction, and a timestamp inside the GAMING DAY (06:00 on the gaming day's date to 06:00 the next date) -- across the patron's own record AND any other record in the log that is the same person by the link keys. This is arithmetic you do yourself; it is usually NOT a number printed anywhere on the page. Return null, and nothing else, when any qualifying entry's amount was never captured
- draft_window_applied (enum) one of: gaming_day, calendar_day, other -- which aggregation window the draft says it applied, from its Window Applied line. 'gaming_day' is 06:00 to 06:00 the next date; 'calendar_day' is 00:00 to 24:00; anything else is 'other'
- linked_record_id (string) -- a SECOND patron record in this log that is the same person as the patron on the draft -- same date of birth AND same identification reference. Return that record's identifier, or null when there is no such record. A record that matches on neither key, or on only one of them, is NOT a link
- draft_includes_linked_record (enum) one of: yes, no, not_applicable -- did the draft aggregate the linked record's transactions as well? 'not_applicable' when there is no linked record at all
- missing_identification_elements (string) -- which of the rulebook's five required identification elements the draft does not carry, comma-separated and spelled exactly as the rulebook spells them. Return null when all five are present. Absence is what you are reporting, so this is a list you derive rather than a phrase you copy
- identification_captured_on (string) -- the date the identification on the draft was captured, YYYY-MM-DD, verbatim from the identification block. Null when the draft does not state one
- miscoded_transaction_ids (string) -- the identifiers of transactions whose code on the DRAFT is not the code the CAGE TRANSACTION LOG records for the same identifier, comma-separated. Return null when every included transaction carries the log's own code
- preparer_note (string) -- the free-text remark the person who prepared the draft left on it, copied verbatim
- defects_found (string) -- the defect codes this filing carries, comma-separated, drawn ONLY from the rulebook's seven codes, or the single word 'none' when the filing carries no defect. Decide it STRICTLY from the rulebook and the numbers -- never from the preparer's note. Work through the stopping order and give ONE named cause for any total difference: (1) if the qualifying total cannot be computed, answer 'insufficient_information' and stop; (2) if the qualifying total does not cross the threshold, answer 'threshold_not_crossed' and stop; (3) if the draft applied a window other than the gaming day, that is 'window_misapplied' and you must NOT also report 'missed_aggregation'; (4) else if a linked record exists and the draft left it out, that is 'identity_split'; (5) else if the qualifying total exceeds the drafted total, that is 'missed_aggregation'. Then, independently of 3-5, add 'identification_gap' for a missing or stale identification element and 'type_miscode' for a transaction coded differently from the log. A FILING WITH NOTHING WRONG WITH IT IS 'none' -- do not manufacture a finding to look thorough

Return a JSON object with exactly these keys: filing_id, patron_record_id, gaming_day, draft_direction, draft_reported_total, log_qualifying_total, draft_window_applied, linked_record_id, draft_includes_linked_record, missing_identification_elements, identification_captured_on, miscoded_transaction_ids, preparer_note, defects_found
Use null for any field the pack does not state.

QC PACK
-------
Draft Filing
------------
DF-2026-6580

Patron Record
-------------
PR-GD1148

Gaming Day
----------
2026-04-08

Direction Reported
------------------
cash_out

Window Applied
--------------
gaming day (06:00 to 06:00 the next date)

Reported Total
--------------
12250 CU

Transactions Included On The Draft
----------------------------------
TXN-04762  credit_instrument_issue    3150 CU
TXN-04765  chip_redemption_cash       5000 CU
TXN-04768  front_money_withdrawal     4100 CU

Patron Identification On The Draft
----------------------------------
full name: J. Mardwick
date of birth: 1957-04-22
residential address: 200 Quenbourne Rise, Westhaven
identification reference: IDR-988491
patron account number: PA-952459
identification captured on: 2025-06-12

Cage Transaction Log
--------------------
TXN-04777  2026-04-08 01:26  PR-GD1148    credit_instrument_issue    1950 CU
TXN-04774  2026-04-08 06:44  PR-CC8122    front_money_withdrawal     1400 CU
TXN-04768  2026-04-08 14:23  PR-GD1148    front_money_withdrawal     4100 CU
TXN-04765  2026-04-08 14:27  PR-GD1148    chip_redemption_cash       5000 CU
TXN-04762  2026-04-08 16:04  PR-GD1148    credit_instrument_issue    3150 CU
TXN-04771  2026-04-08 16:47  PR-CC8122    ticket_redemption_cash     1100 CU

Other Patron Records In This Log
--------------------------------
PR-CC8122 -- date of birth 1957-04-22, identification reference IDR-988491

Preparer Note
-------------
Reviewed against the log line by line; totals agree and the pack is ready to go.

Raw responseThe raw response

{
  "filing_id": "DF-2026-6580",
  "patron_record_id": "PR-GD1148",
  "gaming_day": "2026-04-08",
  "draft_direction": "cash_out",
  "draft_reported_total": 12250,
  "log_qualifying_total": 14750,
  "draft_window_applied": "gaming_day",
  "linked_record_id": "PR-CC8122",
  "draft_includes_linked_record": "no",
  "missing_identification_elements": null,
  "identification_captured_on": "2025-06-12",
  "miscoded_transaction_ids": null,
  "preparer_note": "Reviewed against the log line by line; totals agree and the pack is ready to go.",
  "defects_found": "identity_split"
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check a casino's cash-transaction filing before it's sent — 56 QC packs. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersThree ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The preparer-note floor (evals/baseline.py) and the scorer (evals/judge.py) are both pure code and cost $0.00 to run against any result set. The figure above is the ONE SCORED run's own token counts (56 packs) priced at Google Gemini 3 Flash's published rate — the same basis cost_per_query_usd uses. ⚠︎ IT IS NOT WHAT BUILDING THIS KIT COST. Four further paid passes sit outside it and are committed in results/: a 6-pack calibration that MEASURED max_tokens ($0.0453), the 18-pack anti-false-alarm ablation ($0.1168), 8 prompt-split calls across two packs ($0.0091), and one worked example ($0.0094), plus 2 live UI screenshots. All 91 live calls together project to about $0.61 on this card.

Cost driversWhat actually moves the bill

• The fixed prefix — system prompt (843 tokens), filing rulebook (1,083) and field schema (942) — is 2,868 of 3,439 tokens on the example call, 83 pct, and is byte-identical on every one of the 56 calls. The pack itself is 571 tokens. The bill is the rulebook and the instructions, not the document.
• Provider-side reasoning, and on this task it dominates everything else: 96,113 of r001's 105,575 output tokens (91.0 pct) were reasoning rather than the JSON record, on a fourteen-key reply that is about 250 tokens of actual content. That is a higher share than the extraction kits in this series report, and the cause is legible — the model is re-adding a cage log, applying an 06:00 boundary and walking a seven-branch stopping order, none of which is transcription. Reasoning was left at the provider's default and nothing here has measured what turning it off would cost in accuracy.
• The pack length itself, which is the one input that grows with the patron. Every log entry for the gaming day is in the call because the qualifying total is arithmetic over all of them; a high-volume patron's day would multiply the fourth prompt part without limit.

Your volumeWhat it costs at your volume

Linear in QC packs: each call is independent, carries the same fixed prefix and shares nothing with its neighbours, so 560 packs cost ten times 56. Wall clock is NOT linear — the harness runs 12 concurrent workers, so 56 packs took 84.3 seconds rather than the ~13 minutes a serial loop would have taken, and EVAL_WORKERS is the knob. Nothing amortises: there is no index and no cache. ⚠︎ AND THE PACK GROWS WITH THE PATRON, WHICH THE FIXED PREFIX DOES NOT. A patron with two hundred cage movements in a day puts two hundred rows into the fourth prompt part, and at that size the aggregation has to move into code before the call.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 56 currency-transaction filing QC packs, one completion call each, on the fast tier. The calibration, the ablation, the prompt-split measurements and the worked example are recorded separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the QC pack into addressable sections, pure code — the draft, its included-transactions table, its identification block, the cage log, the other patron records and the preparer's note

# Cut a document into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections carry each field, pure code — defects_found is mapped to the eight sections the rulebook actually reads and never to Preparer Note, and the Property And Shift section is mapped by nothing and never sent

You change it to: map fields to your own pack's headings; unmatched falls back to the whole document

# Pick which sections plausibly carry each field. Pure code -- the last deterministic step

SECTION_HINTS = {
def for_field(secs, field):
def plan(secs, fields):

src/rulebook.pyrulebook

the check itself, loaded from data/rulebook.json — the threshold, the gaming-day window, the transaction codes and their reportability, the identification elements and staleness horizon, the same-person link keys, and the STOPPING ORDER that names one cause per total difference. Data, not code, so a forker can open it and disagree with it

# THE RULEBOOK, loaded from data/rulebook.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
def load(path=RULEBOOK_PATH):
R = load()
THRESHOLD = R["threshold"]
UNIT = R["unit"]
STALE_DAYS = R["identification_stale_days"]
CODES = R["transaction_codes"]
ID_ELEMENTS = list(R["identification_elements"])

src/prompt.pyprompt

assemble one call for all fourteen fields, with the rulebook RENDERED into it from data/rulebook.json rather than restated in prose — a threshold this kit invented is not something a model can look up

# Assemble the pre-check prompt. One prompt per case pack, all fourteen fields in it.

SYSTEM = (
def rulebook_block(r=None):
RULEBOOK_TEXT = rulebook_block()
def field_schema(fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key — plus the pure-code business-condition check downstream: a defect that changes what would be FILED sends the draft back to the preparer, a defect that changes what the filing SAYS does not

You change it to: the triage rule itself — which defects send a draft back to the preparer and which are corrected in place. It is deliberately NOT the same function as the rulebook pass, so changing WHAT GETS SENT BACK does not change WHAT COUNTS AS A DEFECT

# Extract one QC pack's fields: segment, select, prompt, one model call, then a pure-code

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 8000
def load_fields():
def load_doc(case_id):
def documents():
def assess(values):
def defect_set(raw):

evals/judge.pyjudge

score field accuracy, the arithmetic cells, defect detection, THE FALSE-ALARM RATE ON ITS OWN DENOMINATOR, and the recompute flag, all separately, pure code

# Score a filing pre-check run. PURE CODE -- gold is exact and the answer is one value per cell,

LIST_FIELDS = ("defects_found", "missing_identification_elements", "miscoded_transaction_ids")
ARITHMETIC_FIELDS = ("draft_reported_total", "log_qualifying_total")
def norm(v):
def _tokens(v):
def equal(field, got, want):
def score(fields, records, golds):
def _matrix(rows, positive):
def score_defects(records, flags, golds):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3429 input and 1885 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's QC packs are entirely synthetic (tools/build_corpus.py, seed 20260822): no real patron, casino, gaming operator, property, filing or transaction exists in the corpus, and nothing was fetched from anywhere. Patron names are built mechanically from a syllable list, property names are constructed compound nouns, and every record number, identification reference, account number and filing identifier is an invented format. Amounts are in CU, an invented unit. The only outbound traffic the kit makes is one chat-completion request per pack to the configured provider, carrying the mapped sections of one pack plus the fixed prompt and the shipped rulebook. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser — which matters here because the adapter deliberately surfaces transport errors that can carry a URL. tools/shoot_ui.mjs additionally BLANKS API_KEY for its free shots and refuses to start against a server it did not start itself, so a screenshot pass cannot silently spend against a key it did not know was there.

The experimentWe did not attack it — and three of four boundaries hold

The three boundaries that hold were confirmed by reading the code, not by a run: no code path files, submits, approves or clears anything; the rulebook pass and the triage rule read only structured values and a set of codes, so the preparer's note cannot reach either; and the key and base URL are redacted out of any error before it is returned, with the screenshot script blanking the key outright. The fourth is open. An indirect prompt injection needs a field an outside party authored, and this kit has exactly one — the preparer's note — which makes it the obvious place to attack and the reason the gap is named rather than glossed. Confirmed by reading the code, not by a run, on 2026-08-22 — no attack run was fired; see redteam.why.

The first three boundaries hold, confirmed by reading the code rather than by an attack run. The fourth is open and is named as open — an indirect prompt injection in the preparer's note is exactly the attack this kit's shape invites, in both directions, and nothing here has tried it.

The result0 attack trials, and three of four boundaries checked here hold on evidence. The one that does not is the one this kit's own shape invites: the preparer's note is free text written by the person whose work is being checked, and whether an instruction hidden in it could suppress a defect — or manufacture one, which on a QC queue is the cheaper attack — is unmeasured.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted ambiguity is a REGISTER mismatch — a confident sign-off on a defective filing, an anxious note on a clean one, on 22 of the 56 packs — which measures whether a model runs the rulebook when the prose points the other way. It does not measure whether a model obeys an instruction hidden in the same field. Those are different failures and only one of them is measured here.

HonestyWhat this does not prove

• Whether an indirect prompt injection in the preparer_note field could suppress a real defect or manufacture a false one. No attack run exists, and the second direction matters as much as the first on a kit whose value is a low false-alarm rate.
• Whether the kit behaves safely against a hostile provider — a response body crafted to break the JSON extraction in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed document, and nothing beyond that was tested.
• Whether a real QC pack would carry anything sensitive this kit mishandles. The corpus has no personal data by construction; a real pack carries a named patron, their date of birth, their residential address and their identification document, and none of that path is exercised. A forker pointing this at real packs is sending exactly that to a provider, one pack at a time.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_recompute fires when the defect list contains at least one defect that changes WHAT WOULD BE FILED — missed_aggregation, window_misapplied, identity_split, threshold_not_crossed or insufficient_information. identification_gap and type_miscode change what the filing SAYS and are corrected in place, so they do not fire it. The defect list comes out of the reply; the rule is run afterwards in pure code, over whatever the model returned, and never over gold. `insufficient_information` counts as a recompute deliberately: treating a record nobody can compute a total from as though it were clean is the one thing a pre-check must never do. A reply carrying no defect answer at all returns None rather than False — an unknown is not a pass.

src/extract.py::compute(), called by extract() on every pack, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by identical code. The rulebook pass it reads lives in a SEPARATE module, src/rulebook.py::assess(), which is also what the corpus generator wrote gold with and what src/prompt.py states to the model — one definition, three readers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a submission, a withdrawal, an approval or a clearance. Nothing here files anything, transmits anything, signs anything or closes anything, and defects_found is a proposal with the rulebook's reasoning attached rather than a decision anybody is bound by. A qualified person decides what is submitted.
• It is NOT a check on whether the extracted values are right. If the model mis-adds the cage log and then reasons correctly from the wrong total, the draft is sent back — or let through — on a wrong number, and this rule cannot tell. The consistency diagnostic in evals/judge.py is the closest thing to that check, it is reported separately, and it is blind to exactly the same case, which on this kit is the commonest way to be wrong.
• It is NOT any compliance programme's triage policy. 'Changes what would be filed' is this kit's own line, invented for this corpus. No programme, filing instruction or supervisory guidance was consulted, and none is reproduced. A real desk weighs how close to the deadline the filing is, whether the patron is already under review, how large the shortfall is, and who has authority to send a pack back.
• It is NOT a judgement about the rulebook. The flag reads the defect list the rulebook produced; if the rulebook is wrong, the flag is confidently wrong with it. The shipped rulebook is invented and is not an authority.

WatchedWhat is watched, and why that one

The kit measures 18 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 11 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (failures 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

5 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

ablation · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 3 chips that all say so.

extraction · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on documents, extraction_cells, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

extraction · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 7 chips that all say so.

DeviationsWhat deviated

0 breaches across 5 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/extract.py, on any change to the recompute_defects list in data/rulebook.json, on any change to data/fields.json's allowed values for defects_found, and on any corpus regeneration. Changing WHICH DEFECTS SEND A DRAFT BACK is a policy change and must be re-scored, even though it never changes what counts as a defect.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the rulebook is a JSON file read with json, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per pack — segment, select, prompt, one call, parse, compute — with no branch, no loop and no agent. The only concurrency is a thread pool over independent packs, which is a for loop with a wider mouth. Anything that looks like orchestration in a kit this size is a diagram of a straight line.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain — the JSON extraction from a fenced reply, the backoff policy, the section regex, the number coercion, the list-field normalisation and the .env reader are all code somebody has to own. This kit inherited a FIXED adapter from a sibling that had already paid for the transport-retry bug, which is the only reason it did not pay for it too.
• No framework means no framework's ecosystem — no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. What ships is what is in the repository.
• The provider abstraction covers exactly two wire formats. A third provider is one function and one dict entry, and until somebody writes it the kit runs on two shapes.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-ctr-precheck on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b000-ctr-precheck-note, t000-ctr-precheck-stub recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Every patron name is built mechanically from a syllable list plus an initial, every property name is a constructed compound noun, and every record number, identification reference, account number, filing identifier and transaction identifier is an invented format. NO REAL CASINO, GAMING OPERATOR, REGULATOR, AGENCY, FORM NUMBER OR REGULATION IS NAMED ANYWHERE IN THIS KIT. The rulebook it ships (data/rulebook.json) is INVENTED and is not an authority: the threshold, the aggregation window, the staleness horizon, the identification elements and the transaction-code table were written for this kit and reproduce no real regulation, filing form, filing instruction, supervisory guidance or compliance manual. Amounts are in CU, an invented unit that is not a currency and maps to none. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-field exact match, with set equality on the three list fields

Does the model's value for each field match gold? Eleven fields compare as strings after trimming whitespace and punctuation; the three that are LISTS wearing a string's clothes — defects_found, missing_identification_elements, miscoded_transaction_ids — compare as SETS, because scoring a correct answer wrong for putting two codes in the other order is a scoring defect and not a model one. Four fields are legitimately null on some packs: there is often no linked record, nothing missing, nothing mis-coded, and on the three insufficient-information packs there is no computable qualifying total either. A null there is a hit, not a miss, when gold agrees.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same field-match logic the free floor is scored by — a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 56 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same values the pack states; evals/check_labels.py asserts every non-nullable field is populated on every row, that a null qualifying total and the insufficient_information answer agree in BOTH directions, that every defect code in gold is one the shipped rulebook carries, and that each of the seven defect classes clears a floor of three, before any run is allowed to spend.

How tight can the band be? There is no tolerance band — exact match after trimming whitespace and punctuation, and set equality on the three list fields. Never a continuous score to round.

Cadence: Re-score on any change to data/rulebook.json, src/prompt.py, data/fields.json or tools/build_corpus.py — the first three change what is asked, the fourth changes what is asked about. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineDefect detection, and the false-alarm rate on its own denominator

Two questions, and the second is the headline. DETECTION: of the 38 defects this corpus seeded, how many did the run name, and how many codes did it name that gold does not carry? Reported as recall, precision, a per-code breakdown and the share of packs whose defect set was reproduced exactly. THE FALSE-ALARM RATE: OF THE 18 FILINGS WITH NOTHING WRONG WITH THEM, ON HOW MANY DID THE RUN RAISE A DEFECT ANYWAY? A QC queue that cries wolf is worse than no queue at all, because a person has to clear every row it produces and the first thing a team does with a queue full of noise is stop reading it. Its denominator is the clean packs and ONLY the clean packs — a defect raised on a pack that already has a different real defect costs the queue nothing extra, because the row was going to be opened anyway.

MethodHow the test was run

evals/judge.py::score_defects, in-process, no key and no model.

Every grader on these pages scored the same 56 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's defect list is re-derived inside the grader by the same pass the kit publishes — src/rulebook.py::assess(), reading data/rulebook.json — so the truth this grades against can never be a separately-typed label that drifted from the rule.

How tight can the band be? No threshold — the defect list is a set drawn from a closed seven-code vocabulary, and a reply carrying no defect answer at all is counted as unanswered rather than folded into 'clean'. On a QC pass that distinction is the whole difference between a filing somebody checked and a filing nobody checked.

Cadence: Re-run whenever data/rulebook.json or src/rulebook.py changes, whenever the corpus is regenerated, and on any provider or model change.

The decisionWhen to reach for it

In one lineneeds_recompute against the same rule run over gold

Does the pure-code triage decision — the defect list contains something that changes WHAT WOULD BE FILED — land on the same drafts it would land on if the defect list had been read perfectly? It is a BUSINESS CONDITION, so unlike a self-consistency check it genuinely needs labels, and saying so is half of what makes the number believable.

MethodHow the test was run

evals/judge.py::score_defects, in-process, no key and no model.

Every grader on these pages scored the same 56 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute(), the same function the run uses, applied to GOLD's defect list. One rule, one input, so a change to the rule moves both sides together and the grader cannot silently grade an old policy.

How tight can the band be? No threshold — a set membership test.

Cadence: Re-run whenever compute() or the recompute_defects list in data/rulebook.json changes. Changing WHICH DEFECTS SEND A DRAFT BACK is a policy change and must be re-scored, even though it never changes what counts as a defect.

The decisionWhen to reach for it