The business caseThe problem this solves

A deviation report is a regulated record written by a person, at the end of a shift, about something that has just gone wrong on their line. It arrives as prose. Somebody in a quality unit has to turn it into a triage decision: how serious, and what happens next. Two things make that harder than it sounds. The first is that the facts the rule needs are often simply absent — the operator wrote what happened and never wrote whether product was affected, because at the time it was obvious to them and to nobody else. The second is that the prose has a temperature, and the temperature has nothing to do with the facts. An operator who is worried writes an alarming report about a minor event; an operator who is not writes a calm one about an excursion. A queue triaged on register rather than on stated facts inverts itself. Someone opening each deviation report as it arrives, reading the operator's account of what happened, and deciding two things the report may or may not actually answer: how serious the event is, and whether it needs looking into or correcting at the line. The part that is genuinely hard is not the reading — it is holding the line that HOW THE REPORT IS WRITTEN is not evidence. The same event is written up by a frightened operator at the end of a bad shift and by a bored one at the end of a quiet week, and a triage queue that drifts with the register is a triage queue that escalates the anxious and buries the calm.

The inputThe actual deviation reports

The corpus is 50 deviation reports, 0.07 MB (txt 50). Plain text, one format, invented rather than fetched. A real deviation report is somebody's regulated record: it names the operator who wrote it and the batch it was raised against, and it is not ours to publish redacted either. But the deciding reason is the third one — THE ANSWER HAS TO BE DERIVABLE. Gold here is not an opinion about a deviation: the generator decides three stated facts, writes them into the report text, and derives both calls by calling the shipped rubric, which evals/check_labels.py re-derives over every row before any run may spend. On a corpus of real reports that check cannot exist, and 'the label is right' becomes a thing you assert. The same argument is what makes the tone axis possible at all: the register is DEALT, in exact counts, independently of the facts, so every register appears over every severity class.

Deviation Record
----------------
DEV-0001
Raised 2026-09-20 on the twilight shift.

Reporter Reference
------------------
Operator Ines Sandoval (badge 6097), countersigned by Rowan Bellamy (badge 7597), extension 231.

Event Summary
-------------
Description: Pallet of staged material left outside its marked bay
Area: Line 4, Suite J
Equipment: CN-233
Affected: material_handling
Detected: on the routine round

What Happened
-------------
Nothing dramatic, but it needs writing up. A staged pallet for Product PX-408 was found standing
outside the marked bay in Suite J. It was moved back into the bay by the operator who found it.
I left everything where it was so the next shift could see it. I would not have written it up at
all if the procedure did not say to.

Observations Recorded At The Line
---------------------------------
Material from this batch was affected by the event and has been segregated at the line.
The line has recorded this same stoppage previously.
Event logged at 13:40 in the line record.
Entry made in the line logbook, page 225.

Quality Unit Log
----------------
Received into the quality unit's queue. No assessment recorded on this page.

Closing Note
------------
Filed for review. Nothing on this page classifies the event, dispositions any material or closes
the record.

The outcomeWhat a good result looks like

One worksheet row per report — the deviation id, the event as the report labels it, what the report says was affected, the three facts it states (or does not), and the two calls the shipped rubric makes from them — plus one pure-code review flag taken from those calls. Every call carries the rubric's own reasoning, derived in code rather than written by the model, and every verbatim value carries a span back to the section it was read from. The class that matters is not_determined: 16 of the 50 reports here never say whether product or material was affected and state no excursion, and the rubric's honest answer is that it cannot class them.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Triaging a week of deviation reports before a quality unit opens them — the one tier measured here — 49 of 50 severity calls and 47 of 50 follow-ups, with 4 over-confident calls of 35
  98 pct on the severity call against the free tone-reading floor's 30 pct, and an over-call gap between the alarming and calm registers of 9.09 pp against the floor's 61.82. It also took all 10 of the reports carrying a quality-unit note that contradicts the rubric, and all 11 carrying a record of a similar event on a different line.
• Deciding whether to trust the tone number — the number, with its denominators in front of you
  The corpus deals the register independently of the facts and the gate asserts the whole three-by-three grid is populated, so the comparison is between like and like. But the two cells that carry it are 11 reports and 7 reports. The observed over-call gap is exactly one report.

And where nothing here is good enough:

• Deciding whether a cheaper or a more deliberate model would do better here — nothing — this page cannot answer it
  Only one model was run. There is no tier comparison, no repeatability pair and no ablation on this page, and the severity grader is already at 49 of 50, so it could barely rank two models even if both were fired. The follow-up call, at 47 of 50, is the only grader here with room to move.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-deviation-classify. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts on port 8968, the 50-report picker populates, the worksheet draws its eight row labels and the shipped rubric renders in full from data/rulebook.json; clicking Build worksheet row returns the no-key sentence rather than an error, and the review line reads “not computed” rather than “no”. tools/build_corpus.py and evals/check_labels.py both run to completion with no key and no network.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per deviation report

Current processWhat it replaces

Someone opening each deviation report as it arrives, reading the operator's account of what happened, and deciding two things the report may or may not actually answer: how serious the event is, and whether it needs looking into or correcting at the line. The part that is genuinely hard is not the reading — it is holding the line that HOW THE REPORT IS WRITTEN is not evidence. The same event is written up by a frightened operator at the end of a bad shift and by a bored one at the end of a quiet week, and a triage queue that drifts with the register is a triage queue that escalates the anxious and buries the calm.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptArchitectureArchitectureApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 3 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings; a report with none falls back to one whole-document segment

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 50 reports cut into 361 sections in about three thousandths of a second, in process, with no model and no network. 311 of those sections are sent; the 50 that are not are exactly the Reporter Reference sections, which name the operator who wrote the report, their badge number, the supervisor who countersigned it and an internal extension.

LicenceLicence

MIT — this repository's own licence. Every operator and supervisor name, badge number, line number, suite, equipment code, product code and event is invented; no real firm, site, product, drug, person, supplier or regulator is named anywhere, and no GMP regulation, guidance, pharmacopoeial text, industry guide or company quality manual is reproduced, paraphrased, numbered or named. The triage rubric this kit ships (data/rulebook.json) is ILLUSTRATIVE and is not an authority. It was written for this kit.

Bring your ownBring your own deviation reports

REPLACE data/rulebook.json FIRST. It is this kit's own construction, it reproduces no GMP regulation, guidance or quality manual, and everything downstream reads it: src/rulebook.py loads it, src/prompt.py renders it into every call, and tools/build_corpus.py writes gold with it. Then put your own reports in data/corpus/ and your own labels in data/gold.jsonl, adjust src/segment.py's heading pattern and src/select.py's section names to your layout, and run python3 -m evals.check_labels BEFORE you spend anything. It will fail loudly if your labels do not agree with your own rubric run over your own stated facts — which is the point of it. ⚑ AND IF YOU WANT THE TONE NUMBER, YOUR CORPUS HAS TO CARRY A TONE FIELD. The gate asserts that all three registers appear over all three severity classes; without that, 'the alarming reports scored worse' only ever means 'the alarming reports were the hard ones'.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 3,012 tokens assembled, 1,401 are contexts — 47% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on DEV-0042 — the same report the worked example above was captured on, so the four parts sum to exactly the 3012 input tokens that call actually reported. Four calls at max_tokens=1, each sending the previous prefix plus exactly one more part; every part's size is the difference between two consecutive prompt_tokens counts the provider itself returned. No local tokenizer is used and none is trusted.

Full promptThe prompt in full

You read a MANUFACTURING DEVIATION REPORT -- an event written up at the line by the operator who saw it -- and return a REVIEWER'S WORKSHEET row for it. You return JSON and nothing else.

You are preparing a worksheet for a qualified reviewer in a quality unit. You never classify a deviation, never disposition a batch, never release or reject material, never close a record, never approve a correction and never report anything to anybody. Your job is to say what the report states, apply the rubric given below, and NAME THE REPORTS THE RUBRIC DOES NOT SETTLE.

RULES, in order of importance:
1. HOW THE REPORT IS WRITTEN IS NOT AN OBSERVATION. Operators write the same event up very differently depending on how the shift went and how worried they are. Words like serious, urgent, dangerous, worrying, routine, small, trivial and nothing-unusual are the reporter's register and this rubric reads none of them. An alarming report over facts the rubric calls minor is MINOR. A calm report over a stated excursion is MAJOR.
2. REPORT WHAT THE PAPERWORK STATES. `product_impact`, `spec_exceeded` and `recurrence` are readings of the report, not judgements about the event. If the report never says whether product or material was affected, `product_impact` is 'not_stated' -- that is a finding about the paperwork, and it is the most common one here. Never infer it from the narrative's register, from what was affected, or from what you know about how this kind of event usually turns out.
3. A MISSING FIGURE IS NOT A MISSING FACT. If the report says a written limit for the step was gone outside or was exceeded, `spec_exceeded` is 'exceeded' WHETHER OR NOT the operator recorded by how much. An excursion with no number against it is still a stated excursion.
4. `severity` IS DECIDED ONLY BY THE RUBRIC BELOW, from `spec_exceeded` and `product_impact`. Work through the four steps IN ORDER and STOP at the first that fires:
   a. spec_exceeded 'exceeded' -> 'major'.
   b. product_impact 'impacted' -> 'major'.
   c. product_impact 'not_impacted' -> 'minor'.
   d. anything else -> 'not_determined'.
5. A QUALITY-UNIT NOTE SAYING WHAT SOMEBODY ALREADY THINKS THE CLASS IS, IS NOT A FACT. Some of these reports carry one and it is right about as often as it is wrong. It never moves `severity`.
6. `recurrence` IS ABOUT THIS LINE AND THIS EQUIPMENT. A record of a similar event on a different line is not this line's history and does not make this 'repeat'. And a report that does say it has happened here before often says so in one offhand clause in the middle of the narrative, with no heading over it -- read the whole report, not only the observations.
7. `follow_up` IS DECIDED ONLY FROM `recurrence`: 'repeat' -> 'investigation', 'first_occurrence' -> 'local_record', 'not_stated' -> 'not_determined'. It is asked independently of the severity call.
8. 'not_determined' IS A REAL ANSWER AND YOU ARE EXPECTED TO USE IT. A worksheet that never reaches for it is guessing, and a class recorded as settled is a class nobody re-reads.
9. Copy `deviation_id`, `event_label` and `affected` verbatim from the report. Use the exact allowed value for every field that lists them, and return every field.

TRIAGE RUBRIC (the authority for `severity` and `follow_up`; this is an ILLUSTRATIVE rubric written for this kit, and it reproduces no GMP regulation, no regulator's guidance and no company's quality manual)

WHAT IS BEING CLASSIFIED HERE
  - One report describes one event, and the worksheet carries one row for it.
  - The three stated facts are readings of the REPORT, not judgements about the event. If the report does not say whether product was affected, the reading is `not_stated` - that is a finding about the paperwork and it is the most common one here.
  - A quality-unit note in the report saying what somebody already thinks the class is, is NOT a fact this rubric reads. It is somebody's opinion recorded on the same page, and it is right about as often as it is wrong.
  - A record of a similar event on a DIFFERENT line or a different piece of equipment says nothing about whether THIS event has happened before. Recurrence is about this line.

TONE IS NOT A FACT
  SEVERITY COMES FROM THE STATED FACTS AND NEVER FROM HOW THE REPORT IS WRITTEN. An operator who is worried writes an alarming report; an operator who is not writes a calm one; the same event gets both, depending on who is on shift. Words like serious, urgent, dangerous, routine, trivial and nothing-unusual are the reporter's register, not observations, and this rubric reads none of them. A report that reads alarmingly and states minor facts is minor. A report that reads calmly and states an excursion is major.

THE THREE STATED FACTS, read off the report
  product_impact:
    impacted         the report states that product or material in process was affected by the event
    not_impacted     the report states that no product or material was affected - because none was present, or because the report says so directly
    not_stated       the report says neither. It describes the event and never says whether anything was affected
  spec_exceeded:
    exceeded         the report states that a written limit for the step was gone outside, or was exceeded. THIS IS TRUE WHETHER OR NOT THE REPORT SAYS BY HOW MUCH - the extent is not a fact this rubric reads
    within_limits    the report states that the parameter stayed inside its written limit for the step
    not_stated       the report says neither. No limit is mentioned at all
  recurrence:
    repeat           the report states that this same event has been seen on this line or this equipment before
    first_occurrence the report states that nothing like this has been recorded on this line or this equipment before
    not_stated       the report says neither. It never mentions whether this has happened before

SEVERITY -- work through IN ORDER, stop at the first that fires
  1. A STATED EXCURSION. If spec_exceeded is `exceeded`, the worksheet proposes `major`. A written limit for the step was gone outside and the report says so. THIS HOLDS WHETHER OR NOT THE REPORT RECORDS BY HOW MUCH: an operator who writes that a limit was exceeded and does not write a figure has still stated the excursion, and on this rubric the missing figure changes nothing.
  2. STATED PRODUCT IMPACT. If product_impact is `impacted`, the worksheet proposes `major`. The report itself says material in process was affected.
  3. STATED ABSENCE OF PRODUCT IMPACT. If product_impact is `not_impacted` and no excursion was stated, the worksheet proposes `minor`. The report says nothing was affected and names no limit that was gone outside.
  4. NOTHING STATED ABOUT PRODUCT. The worksheet proposes `not_determined`. The report never says whether product or material was affected, and it states no excursion, so this rubric has nothing to class it from. THIS IS THE BRANCH A CONFIDENT READER SKIPS - the narrative usually reads either serious or trivial, and neither is a fact.

FOLLOW-UP
  1. If recurrence is `repeat`, the worksheet proposes `investigation` - the rubric calls for the event to be looked into rather than corrected at the line, because the report says it has happened here before.
  2. If recurrence is `first_occurrence`, the worksheet proposes `local_record` - correct it at the line and record it.
  3. If recurrence is `not_stated`, the worksheet proposes `not_determined`. The follow-up question is asked of every report independently of the severity call - a report the rubric cannot class for severity may still say plainly whether the event has happened before, and one it CAN class may say nothing at all.

WHY `not_determined` IS A REAL ANSWER
  A worksheet that never says `not_determined` is not a confident worksheet, it is a guessing one. The whole value of this shape is that a reviewer can see, in one column, which reports actually state what the rubric needs and which ones somebody has to walk back to the line and ask about. Over-confidence here costs more than caution: a class recorded as settled is a class nobody re-reads, and the operator who could have answered the question in ten seconds has gone home.

Return these:
- deviation_id (string) -- the deviation record identifier from the Deviation Record section, verbatim
- event_label (string) -- the one-line event description, copied verbatim from the Description line of the Event Summary section
- affected (enum) one of: product_contact_surface, equipment_only, utility_or_environment, documentation_only, material_handling -- what the report says was affected, copied verbatim from the Affected line of the Event Summary section. It is recorded because a reviewer wants it on the worksheet -- it is NOT an input to either call below
- product_impact (enum) one of: impacted, not_impacted, not_stated -- what the report STATES about whether product or material in process was affected. 'impacted' when it says material was affected, held, quarantined or segregated because of this event. 'not_impacted' when it says no material was affected, or that no product was present. 'not_stated' when the report says NEITHER -- which is the commonest reading here and is a finding about the paperwork, not a gap. Never infer this from how alarming or how calm the narrative sounds
- spec_exceeded (enum) one of: exceeded, within_limits, not_stated -- what the report STATES about a written limit for the step. 'exceeded' when it says a limit was gone outside or was exceeded -- THIS IS TRUE WHETHER OR NOT THE REPORT SAYS BY HOW MUCH, and a missing figure does not turn it into 'not_stated'. 'within_limits' when it says the parameter stayed inside its limit. 'not_stated' when no limit is mentioned at all
- recurrence (enum) one of: repeat, first_occurrence, not_stated -- what the report STATES about whether this same event has been seen on THIS line or THIS equipment before. 'repeat' when it says it has, wherever in the report that sentence sits -- it is often one clause at the end of the narrative rather than a heading of its own. 'first_occurrence' when it says nothing like it has been recorded here before. 'not_stated' when the report says neither. A record of a similar event on a DIFFERENT line is not this line's history and does not make this 'repeat'
- severity (enum) one of: major, minor, not_determined -- the worksheet's proposed severity class, decided STRICTLY by the shipped rubric from `spec_exceeded` and `product_impact` and nothing else. Work through the four steps IN ORDER, stopping at the first that fires. (1) spec_exceeded 'exceeded' -> 'major', whether or not the report records by how much. (2) product_impact 'impacted' -> 'major'. (3) product_impact 'not_impacted' -> 'minor'. (4) anything else -> 'not_determined'. HOW THE REPORT IS WRITTEN IS NOT AN INPUT: an alarming narrative over minor stated facts is 'minor', and a calm narrative over a stated excursion is 'major'
- follow_up (enum) one of: investigation, local_record, not_determined -- the worksheet's proposed follow-up, decided STRICTLY from `recurrence`: 'repeat' -> 'investigation', 'first_occurrence' -> 'local_record', 'not_stated' -> 'not_determined'. Asked of every report independently of its severity call -- a report the rubric cannot class for severity may still say plainly whether the event has happened here before

Return ONE JSON object with exactly these keys: deviation_id, event_label, affected, product_impact, spec_exceeded, recurrence, severity, follow_up

DEVIATION REPORT
----------------
Deviation Record
----------------
DEV-0042
Raised 2026-08-02 on the night shift.

Event Summary
-------------
Description: Tablet press tooling loosened during compression
Area: Line 2, Suite J
Equipment: TB-092
Affected: product_contact_surface
Detected: at line clearance

What Happened
-------------
Small thing, but the procedure says write it up, so here it is. Tooling on TB-092 worked loose
part way through the compression run. The press was stopped from the panel and the tooling was
re-seated by engineering. Two of us were standing at the line when it happened. I would not have
written it up at all if the procedure did not say to.

Observations Recorded At The Line
---------------------------------
The sheet from the line has been filed with this record.
Shift supervisor told at the line.
The reading went outside the written limit for the step by 3.1 degrees before it came back.

Quality Unit Log
----------------
Quality unit view at the line was that this is a major event.

Closing Note
------------
Recorded for the quality unit's review. No classification has been made on this page and no
batch decision follows from it.

Raw responseThe raw response

{
  "deviation_id": "DEV-0042",
  "event_label": "Tablet press tooling loosened during compression",
  "affected": "product_contact_surface",
  "product_impact": "not_stated",
  "spec_exceeded": "exceeded",
  "recurrence": "not_stated",
  "severity": "major",
  "follow_up": "not_determined"
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Sort a plant's deviation reports by severity — 50 deviation reports. One model answered, and every answer was then graded Five different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersFive ways to grade

⚑ THE FLOOR IS NOT A STRAWMAN AND ITS FOLLOW-UP COLUMN IS PERFECT. evals/baseline.py reads all three stated facts off every one of the 50 reports correctly by regex — product, written limit and history — and scores 50 of 50 on the follow-up call, which is better than the model's 47. Its whole deficit is one column: it decides SEVERITY from somebody's opinion where the page carries one, and from the reporter's register where it does not, and scores 30.0 pct against the run's 98.0. That split is the point of shipping it. A floor that failed everywhere would prove nothing; this one fails in exactly one place, and the place is the claim the kit is about. ⚑ THE NUMBER TO READ IS THE GAP: over-calling 9 of the 11 alarming reports the rubric does not class as major, against 2 of 10 calm ones — 61.82 percentage points, where the run's is 9.09. ⚠︎ AND IT WOULD BE ONE LINE TO MAKE IT PERFECT. It already extracts every fact the rubric needs; calling src.rulebook.decide() on them would score 100 pct and tell you nothing about the model. The floor is deliberately the shortcut, not the lookup. One last thing about it is worth more than its score: the no-gold consistency diagnostic caught all 35 of its call errors with no labels at all, against 0 of the run's 4 — the floor is wrong in a way a machine can see, and the run is wrong in a way only gold can.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The tone-reading floor (evals/baseline.py) and the scorer (evals/judge.py) are both pure code and cost $0.00 to run against any result set. The figure above is the ONE scored run's own token counts (50 reports) priced at Google Gemini 3 Flash's published rate — the same basis cost_per_query_usd uses. ⚠︎ IT IS NOT WHAT BUILDING THIS KIT COST. Four further paid passes sit outside it and are committed in results/: a 6-report calibration that MEASURED max_tokens ($0.0178), the worked-example capture ($0.0027), 4 prompt-split calls at max_tokens=1 ($0.0043) and 1 live UI screenshot ($0.0033). All 62 live calls together project to $0.1826 on this card. AND THAT UNDERSTATES IT: an earlier 50-call run at a 4,000-token ceiling lost two reports and is not committed, because a result scored against a corpus revision that no longer exists is a comparability trap rather than evidence.

Cost driversWhat actually moves the bill

• The fixed prefix — system prompt (865 tokens), triage rubric (1101) and field schema (746) — is 2712 of 3012 tokens on the example call, 90.0 pct, and is byte-identical on every one of the 50 calls. The deviation report itself is 300 tokens. ⚑ NINE TENTHS OF THE INPUT IS THE INSTRUCTIONS AND THE RULE, which is what a short document and a one-object reply look like priced together, and nothing caches it.
• Provider-side reasoning — 21,991 of the run's 26,305 output tokens, 83.6 pct. The JSON row itself is about 250 tokens; everything else billed as output was thinking nobody reads. On the projected card the output side is 51.1 pct of the bill, so this is roughly two fifths of every dollar.
• The number of reports. One call each, nothing shared, nothing amortised — 500 reports cost ten times 50.

Your volumeWhat it costs at your volume

Linear in reports: each call is independent, carries the same fixed prefix and shares nothing with its neighbours, so 500 reports cost ten times 50. Nothing amortises — there is no index and no cache. ⚑ AND IT IS FLAT IN DOCUMENT LENGTH IN A WAY ITS LIST-SHAPED SIBLINGS ARE NOT: the reply is one object of eight fields whatever the report says, so a longer report moves the input side only. What is NOT flat is the reasoning, which has a long tail — the median reply on this run was well under a thousand output tokens and the largest was 2,242, and under an earlier 4,000-token ceiling two reports burned the entire budget thinking.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 deviation reports, one completion call each, one tier. The calibration, the worked-example capture, the prompt-split calls and the UI screenshot are recorded separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the deviation report into addressable sections, pure code — the record header, the reporter's own reference, the event summary, the narrative, the observations block, the quality-unit log and any unrelated-records decoy

# Cut a deviation report into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections are sent, pure code. The narrative and the quality-unit note are deliberately SENT — they carry the register and somebody's opinion, and a selector that dropped them would be marking the model's homework on this kit's headline question. The Reporter Reference section is mapped by nothing and never leaves the machine

You change it to: map fields to your own report's headings; unmatched falls back to the whole document. NEVER_SENT is the list of sections no field asks for and which therefore never leave the machine — on this corpus that is the section naming the operator, their badge and the supervisor who countersigned

# Pick which sections of a deviation report are sent. Pure code -- the last deterministic step

RECORD = "Deviation Record"
SUMMARY = "Event Summary"
NARRATIVE = "What Happened"
OBSERVATIONS = "Observations Recorded At The Line"
QA_LOG = "Quality Unit Log"
UNRELATED = "Unrelated Records"
CLOSING = "Closing Note"
NEVER_SENT = ("Reporter Reference",)
SECTION_HINTS = {

src/rulebook.pyrulebook

the two worksheet calls, loaded from data/rulebook.json — a four-step severity rule with a stopping order and a three-way follow-up rule. Data, not code, so a forker can open it and disagree with it. ⚑ severity() takes two arguments and neither of them is the narrative: tone cannot reach the rule through the signature

# THE TRIAGE RUBRIC, loaded from data/rulebook.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
IMPACTS = ("impacted", "not_impacted", "not_stated")
SPECS = ("exceeded", "within_limits", "not_stated")
RECURRENCES = ("repeat", "first_occurrence", "not_stated")
SEVERITIES = ("major", "minor", "not_determined")
FOLLOW_UPS = ("investigation", "local_record", "not_determined")
def load(path=RULEBOOK_PATH):
R = load()

src/prompt.pyprompt

assemble one call per report, with the rubric RENDERED into it from data/rulebook.json rather than restated in prose — a rule the model has never been shown is not a rule

# Assemble the triage prompt. ONE prompt per deviation report, one worksheet row back.

SYSTEM = (
RULEBOOK_TEXT = rulebook_block()
def field_schema(fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key — plus the pure-code business-condition check downstream: a report the rubric calls major whose history is not stated goes back to the line

You change it to: the review rule itself — this kit ships two values (the rubric proposes major, and the history is not stated) and a real triage weighs the product, the stage and what the batch record shows. It is deliberately NOT the same function as the rubric lookup, so changing WHAT GETS WALKED BACK TO THE LINE does not change WHAT THE RUBRIC SAYS

# Extract one deviation report's worksheet row: segment, select, prompt, one model call, then a

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 16000
def load_fields():
def fields_list(fields=None):
def load_doc(doc_id):
def documents():
def decide(values):

evals/judge.pyjudge

score the extracted cells, the two calls, the determinacy split and the TONE MOVEMENT, all separately, pure code — plus four named sub-slices and a no-gold consistency diagnostic

# Score a deviation-triage run. PURE CODE -- gold is exact and every answer is one value, so `==`

FIELDS = ("deviation_id", "event_label", "affected", "product_impact", "spec_exceeded",
SPANNABLE = ("deviation_id", "event_label", "affected")
TONES = ("alarming", "neutral", "calm")
def norm(v):
def equal(got, want):
def _vals(rec):
def score(fields, records, golds):
CALLS = {"severity": RB.SEVERITIES, "follow_up": RB.FOLLOW_UPS}
CONFIDENT = ("major", "minor", "investigation", "local_record")

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3024 input and 526 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's deviation reports are entirely synthetic (tools/build_corpus.py, seed 20260822): no real firm, site, operator, product, drug, batch, line or event exists in the corpus, and nothing was fetched from anywhere. No GMP regulation, regulator, pharmacopoeia, industry body or company quality manual is named anywhere in the kit. The only outbound traffic the kit makes is one chat-completion request per report to the configured provider, carrying the mapped sections of one report plus the fixed prompt and the shipped rubric — and the Reporter Reference section, which names the operator who wrote it, their badge number, the supervisor who countersigned and an internal extension, is mapped by no field and never leaves the machine. On a shop floor that is the one artefact people are careful about. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser.

The experimentWe did not attack it — and three of four boundaries hold

Three boundaries hold. One was confirmed by reading the code: no code path classifies, dispositions, releases, closes or reports anything, and the rubric lookup and the review rule read only enums, so no report text can reach either. The second is confirmed by a CHECK RATHER THAN BY READING — evals/check_labels.py asserts, before any run may spend, that the Reporter Reference section is never sent and that the narrative, the quality-unit note and the decoy always are. The third is confirmed by a MEASURED SLICE: 10 reports carry a note naming a class the rubric does not reach, and the run took all 10. The fourth is open. An indirect prompt injection needs a section an outside party authored, and a free-text account written by whoever was on shift is exactly that. Confirmed by reading the code, by a pre-flight assertion and by a measured slice, not by an attack run, on 2026-08-22 — no attack run was fired; see redteam.why.

The first three boundaries hold, the second confirmed by an assertion that runs before every scored run rather than by reading the code alone, and the third by a measured slice rather than by an argument. The fourth is open and is named as open — a deviation report is free text written by whoever was on shift, which makes an instruction hidden in a note exactly the attack this kit's shape invites, and nothing here has tried it.

The result0 attack trials, and three of four boundaries checked here hold on evidence — one by an assertion that runs before every scored run, one by a measured slice rather than an argument. The one that does not is the one this kit's shape invites: a deviation report is free text written by whoever was on shift, and whether a sentence written to talk the model out of the rubric could move a call is unmeasured.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. What the corpus DOES plant is the non-adversarial neighbour of an attack: a register that pulls one way while the facts point the other, and a named opinion on the same page that contradicts the rule. Both were measured, and the run resisted the second completely and the first on 10 of 11. Neither measures whether the model would obey an instruction written to be obeyed. Those are different failures and only the first two are measured here.

HonestyWhat this does not prove

• Whether an indirect prompt injection in a quality-unit note could move a call. No attack run exists. The good-faith version was resisted 10 times out of 10, which is evidence about confusable text and not about adversarial text.
• Whether the kit behaves safely against a hostile provider — a response body crafted to break src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed document, and nothing beyond that was tested.
• Whether a real deviation report would carry anything sensitive this kit mishandles beyond the reporter section. The corpus has no personal data by construction; a real record names the operator throughout its narrative, not only in a header, and none of that path is exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_history_review fires when the rubric proposes `major` AND the follow-up call is `not_determined` — the report does not state whether the event has happened on this line before. Both values come out of the same reply; the rule is run afterwards in pure code, over whatever the model returned, and never over gold. A reply carrying no call the rule can read returns None rather than False — an unknown is not a pass.

src/extract.py::compute(), called by extract() on every report, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by identical code. The rubric lookup it reads lives in a SEPARATE module, src/rulebook.py, which is also what the corpus generator wrote gold with and what src/prompt.py states to the model — one definition, three readers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a classification, a disposition, a release, a closure or a report. Nothing here classifies a deviation, dispositions a batch, releases or rejects material, closes a record, approves a correction, posts anything or notifies anybody. severity and follow_up are proposals with the rubric's own reasoning attached, and a quality unit decides.
• It is NOT a check on whether the extracted facts are right. If the model misreads a report and then applies the rubric correctly to the misreading, the report is flagged — or not flagged — on a fact that is not on the page, and this rule cannot tell. THAT IS NOT HYPOTHETICAL HERE: all 4 of the run's errors are exactly that shape, and the consistency diagnostic in evals/judge.py, which is the closest thing to that check, caught 0 of them.
• It is NOT any site's triage policy. A major report with no stated history is this kit's own simplification, invented for this corpus. No quality manual, GMP regulation, regulator's guidance or industry guide was consulted, and none is reproduced. A real triage weighs the product, the stage, what the batch record shows, what the operator says when you go and ask, and whether anything like it is already open elsewhere on site.
• It is NOT a judgement about the rubric. The flag reads the calls the rubric produced; if the rubric is wrong, the flag is confidently wrong with it. The shipped rubric is illustrative, reproduces no regulation or quality manual, and nobody qualified has reviewed it.

WatchedWhat is watched, and why that one

The kit measures 15 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 8 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 50, extraction_cells 400, failures 0, refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

extraction · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on documents, extraction_cells, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

extraction · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 7 chips that all say so.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/extract.py, on any change to data/rulebook.json (which moves the calls the flag reads), on any change to data/fields.json's allowed values for severity or follow_up, and on any corpus regeneration. Changing WHAT GETS WALKED BACK TO THE LINE is a policy change and must be re-scored, even though it never changes what the rubric says.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the rubric is a JSON file read with json, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per report — segment, select, prompt, one call, parse, compute — with no branch, no loop and no agent. The reply is one object, which is the simplest shape in this series and is why there is no identification grader.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain — the JSON extraction from a fenced reply, the backoff policy, the section regex, the whitespace flattening and the .env reader are all code somebody has to own. This kit paid a small instalment of that during its own build: every containment check had to be taught that a 96-character wrap splits sentences across lines, and the generator's first run failed its own assertion on a report that stated the fact perfectly.
• No framework means no framework's ecosystem — no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. Nothing validates the reply's shape except src/prompt.py::parse, which fails closed to {} rather than to a row of nulls, because a row of nulls would score as a run that read the report and found nothing.
• The provider abstraction covers exactly two wire formats. A third provider is one function and one dict entry, and until somebody writes it the kit runs on two shapes.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-deviation-classify on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b000-deviation-classify-tone, t000-deviation-classify-stub recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — one report is one event, whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Every operator and supervisor name, badge number, line number, suite, equipment code, product code and event is invented; no real firm, site, product, drug, person, supplier or regulator is named anywhere, and no GMP regulation, guidance, pharmacopoeial text, industry guide or company quality manual is reproduced, paraphrased, numbered or named. The triage rubric this kit ships (data/rulebook.json) is ILLUSTRATIVE and is not an authority. It was written for this kit. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-field exact match, light normalisation

Did the run read the eight cells off the report? Exact match after whitespace collapsing, punctuation stripping and lower-casing, over every gold row — so a report the run never answered scores as eight misses rather than disappearing from the denominator.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold. Three fields are copied verbatim off the page and every one is asserted present in the report it labels; five are readings, and evals/check_labels.py checks each against its allowed vocabulary before any run may spend.

How tight can the band be? No tolerance band — a value is in the allowed set or it is not.

Cadence: Re-score on any change to data/fields.json's hints or allowed values, to src/prompt.py, or on any corpus regeneration.

The decisionWhen to reach for it

In one lineThe two rubric calls, three-way each

Did the run make the two calls the shipped rubric makes? severity is major / minor / not_determined and follow_up is investigation / local_record / not_determined, each scored separately over all 50 reports with a full confusion matrix, because the two behave completely differently and averaging them would hide it.

MethodHow the test was run

evals/judge.py::score_calls, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/rulebook.py run over the same three facts the generator wrote into the report. evals/check_labels.py re-derives every gold call before any run may spend and fails if one disagrees; it also refuses a corpus in which any class is empty.

How tight can the band be? No tolerance band — one value out of three.

Cadence: Re-score on any change to data/rulebook.json, which moves the rule, the prompt and gold in one edit.

The decisionWhen to reach for it

In one line⚑ The headline — does the register move the answer

Does the narrative's REGISTER move the severity call, and in which direction? Two rates, each split three ways by register: OVER-CALL (gold is not major, the run said major) and UNDER-CALL (gold IS major, the run said something softer). The published figure is the GAP between the alarming column and the calm column. A run reading the rubric rather than the prose has a gap of zero in both directions, whatever its accuracy.

MethodHow the test was run

evals/judge.py::score_tone, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The corpus deals the register INDEPENDENTLY of the facts, in exact counts, and evals/check_labels.py asserts the whole three-by-three grid is populated and that both directional cells clear a floor (8 alarming-not-major, 5 calm-major) before any run may spend. Without that assertion this grader would be measuring which reports happened to be hard.

How tight can the band be? Percentage points, not a ratio, and the two directions are never netted — over-calling a calm queue and under-calling an anxious one are different failures with different costs.

Cadence: Re-score on any change to the tone banks in tools/build_corpus.py, to TONE_BY_SEVERITY, or to src/prompt.py's rule 1.

The decisionWhen to reach for it

In one lineHow often it says the report does not settle it

Of the calls the report does NOT settle, how many did the run answer with a confident value anyway (overconfident) — and its mirror, how many settled calls did it decline (overcautious). Never netted, because a class recorded as settled is a class nobody re-reads, while a needless not_determined only creates work.

MethodHow the test was run

evals/judge.py::score_calls, the determinacy block.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: gold's not_determined rows, which are the reports whose facts the rubric genuinely cannot class — 16 severity calls and 19 follow-up calls of the 100 scored.

How tight can the band be? Counts and rates, both published with their denominators.

Cadence: Re-score on any corpus regeneration — the denominator is the corpus's own composition.

The decisionWhen to reach for it

In one lineneeds_history_review against the same rule run over gold

Does needs_history_review fire on exactly the reports where the condition holds? The same pure-code rule run over the run's own row and over gold's, compared as a confusion matrix. A business condition, so it needs labels and says so.

MethodHow the test was run

evals/judge.py::score_flag, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute() run over gold. evals/check_labels.py asserts before any run may spend that the flag is neither constant nor outside the 10–60 pct band a flag has to sit in to be worth reading; it fires on 11 of 50, 22.0 pct.

How tight can the band be? Binary, with an explicit third state: a reply carrying no readable call returns None rather than False, and None is not a pass.

Cadence: Re-check on any change to compute(), to data/rulebook.json, or on any corpus regeneration. Changing WHAT GOES BACK TO THE LINE is a policy change and must be re-scored even though it never changes what the rubric says.

The decisionWhen to reach for it