The business caseThe problem this solves

A restricted waste line may only move under a notification that actually accompanies the shipment and actually covers it. Checking that means holding three documents against each other per line — the manifest row, the notice it cites, and the roster saying which notices are travelling — and the commonest way to get it wrong is to see a perfectly good notice printed in the pack and stop reading. Reading one manifest line, the notice it cites and the pack's attached-notices roster together, four times per line, and writing down which of the four facts the paperwork actually settles

The inputThe actual shipment packs

The corpus is 50 shipment packs, 0.20 MB (txt 50). A real waste manifest and a real restriction notification cannot be published: they name a real generator, transporter and receiving facility, they carry account numbers and site references both sides treat as commercial, and the classification and the certification on them are regulated representations somebody signed. There is no public corpus of (manifest, notification, correct worksheet) triples. Generating it also makes the label MECHANICAL: gold's two calls are the shipped rulebook's own lookup over the four comparisons the pack states, never anyone's reading of what a load really is.

Shipment
--------
LDR-0001
Dispatched: 2026-07-19

SYNTHETIC RECORD. This pack was generated for a test kit. It is not a real manifest, not a
real restriction notice and not a real shipment, and no real generator, transporter,
receiving facility, site or waste stream is named in it. The restricted-waste codes and the
treatment standards used below are invented for this kit and are not any real code list.

Generator Account
-----------------
Thorne Valley Chemicals (batch-campaign generator), generator account GA-67983, site
reference ST-8293, billing contact BC-8162, transporter account TA-7013.

Manifest
--------
Manifest MN-68746, dispatched 2026-07-19. Lines as manifested:

  WL-2558   Spent pickling acid from the strip line    RC-CYN  3 x tote_poly
  WL-7454   Baghouse dust from metal recovery          RC-SLV  12 x drum_steel
  WL-8806   Reactive residue from the pilot reactor    RC-MTL  8 x tote_poly
  WL-6584   Oil-water separator sludge                 RC-SLV  1 x drum_steel

Attached Notices
----------------
The following restriction notices accompany this shipment:
  RN-2777
  RN-6203
  RN-6705
Any other Notice block printed in this pack is ON FILE for this generator and does NOT
accompany this shipment.

Notice RN-6203
--------------
Effective date: 2025-12-02
Status: current
Codes covered: RC-OIL, RC-MTL, RC-ACD
Treatment standard certified: TS-STAB
Certification signed: yes

Notice RN-6705
--------------
Effective date: 2026-07-02
Status: current
Codes covered: RC-SLG, RC-OIL, RC-SLV
Treatment standard certified: TS-NEUT
Certification signed: yes

Notice RN-6688
--------------
Effective date: 2026-03-21
Status: current
Codes covered: RC-SLV, RC-ACD, RC-CYN
Treatment standard certified: TS-DEACT
Certification signed: yes

Notice RN-2449
--------------

The outcomeWhat a good result looks like

A worksheet naming, per line, whether the notification holds up, whether the certification is signed, and which lines the paperwork does not settle — with the two directions of being wrong counted separately and never averaged.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Working a week of inbound restriction paperwork before a qualified reviewer opens it — the one tier measured here — 273 of 275 on the restriction call, with 0 false clears on 110 gold gap lines
  99.27 pct on the restriction call against the free floor's 41.82 pct, and — the figure that matters — 0 false clears against the floor's 70 of 110. It also left every one of the 39 decoy codes off, against the floor's 13 phantom struck lines, and its query flag is 50 of 50 against the floor's 28 with 14 false alarms.
• Deciding whether to trust the not_determined column at all — the column, with its cause in front of you
  All 90 open restriction calls have ONE cause and it is worth naming: a current, attached notice that covers the code, on a line stating no treatment standard. Nothing disagrees and nothing has been checked. The run reported 88 of them correctly and declined none of the 185 calls it should have settled.
• Deciding whether you need a model here at all — read the free floor's column before you decide
  The floor is $0.00, needs no key and reads every stated fact off the pack perfectly. On a corpus where the check were genuinely one question, it would be the right answer and this page would say so. It is not: it produces 70 false clears on 110 gold gap lines because the check is THREE documents, and its own stated-facts column contradicts its own call on screen. The floor is shipped as a mode of the local app precisely so that can be seen rather than argued about.

And where nothing here is good enough:

• Deciding whether a cheaper or a more deliberate model would do better here — nothing — this page cannot answer it, and it says so rather than implying a recommendation
  Only one tier was run. There is no tier comparison, no repeatability pair and no ablation on this page. What the corpus CAN still separate is worth saying: it is not saturated — the run made 3 call errors and left 2 lines over-confident, so a second tier would have something to be wrong about, unlike several sibling kits whose graders are all at 100 pct.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-25. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-25 — r001-disposal-restrict. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — git clone, then python -m evals.check_labels and python -m evals.run --run-id b000-disposal-restrict-notice-present --baseline both work with no key, no install and no network — requirements.txt is empty and the corpus and every run record are committed. python -m src.app renders the whole UI offline, including the free-code-floor mode, which answers a real pack with no provider at all.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per shipment pack, 50 packs at 8 concurrent workers; 95.4 seconds wall clock for the whole run

Current processWhat it replaces

Reading one manifest line, the notice it cites and the pack's attached-notices roster together, four times per line, and writing down which of the four facts the paperwork actually settles

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusEvalsEvalsArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 2 of 9 stations carry a recorded failure

scroll → 9 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings; a pack with none falls back to one whole-document segment. Nothing is chunked for retrieval — there is no retrieval

No train/test split — nothing is trained and nothing is fitted. All 50 packs are scored, and the same 50 are read by the free floor and by the stub, so the three arms are directly comparable. The composition is an exact JOINT table of (restriction bucket, certification bucket) pairs, dealt and shuffled, not two independent margins.

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 50 packs cut into 882 sections in about nine thousandths of a second, in process, with no model and no network. 832 of those sections are sent; the 50 that are not are exactly the Generator Account sections, which carry the generator's name, what kind of site it is, a generator account number, a site reference, a billing contact and the transporter's account, and which no field asks for. evals/check_labels.py asserts that over all 50 packs before any run may spend.

LicenceLicence

MIT — this repository's own licence

Bring your ownBring your own shipment packs

Drop your own packs into data/corpus/ as .txt and point data/fields.json and data/rulebook.json at your own vocabulary — those two files are the whole domain. If your packs use different section headings, edit SECTION_HINTS in src/select.py; if they use different row labels, edit the regexes in evals/baseline.py and tools/build_corpus.py. To score, write a gold.jsonl with one row per pack carrying its manifested lines and its decoy codes, then run python -m evals.check_labels — it will tell you what does not line up before you spend anything.

⚠︎ And what stops being true when you do: What does NOT port: the gold labels, the composition table and every floor in check_labels.py are properties of THIS corpus. The rulebook is illustrative and its code schemes are invented, so a real facility replaces it wholesale rather than extending it — and the moment it does, the published numbers on this page describe a different question.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 5,153 tokens assembled, 1,886 are rules — 37% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The exact prompt sent for LDR-0002 on the worked-example capture (results/example-LDR-0002.json), byte-identical in shape to all 50 calls of run r001-disposal-restrict — the system prompt, rulebook and field schema are the same on every call and only the pack sections differ.

Full promptThe prompt in full

You read a WASTE SHIPMENT PACK -- a manifest of restricted lines, the disposal-restriction notices printed with it, and the roster saying which of those notices actually accompany the shipment -- and return a REVIEWER'S WORKSHEET, line by line. You return JSON and nothing else.

You are preparing a worksheet for a qualified reviewer. You never authorise a shipment, never accept or reject a load, never clear a line for transport, never sign or countersign a certification, never amend a notice and never file anything with anybody. Your job is to say what the paperwork states, apply the rulebook given below, and NAME THE LINES WHOSE RESTRICTION NOTIFICATION DOES NOT HOLD UP.

RULES, in order of importance:
1. ONE OBJECT PER MANIFESTED LINE OF THIS SHIPMENT, AND NO OTHERS. A line struck from the manifest before dispatch is not manifested, even though its code, its description and its whole Line block are still printed in the pack. A line on the generator's approved shipping schedule with nothing shipped under it is not a line -- a schedule is an approval, not a shipment. A line carried on an EARLIER shipment under the same notice is not a line either. Putting any of the three on the worksheet puts a shipment in front of a reviewer that nobody made.
2. A NOTICE BEING PRINTED IN THIS PACK IS NOT EVIDENCE THAT IT ACCOMPANIES THE SHIPMENT. Read the `Attached Notices` roster first. It names the notice codes that travel with this load. Every other `Notice RN-nnnn` block in the pack is ON FILE for this generator only, and it is printed in exactly the same layout, under exactly the same kind of heading, with exactly the same rows. A line citing one of those is `on_file_only`, and its restriction call is `gap` no matter how perfectly the notice itself reads. THIS IS THE MISTAKE THIS WORKSHEET EXISTS TO CATCH.
3. EVERY READING IS AGAINST THE NOTICE THAT LINE CITES, and no other notice in the pack. Read the line's 'Restriction notice cited on this line' row first, find that Notice block, and read against it. If the Line block has no such row, the line cites nothing: `notice_status` is 'none_cited' and the other three facts are 'no_notice_to_read'.
4. REPORT WHAT THE PACK STATES. `notice_status`, `code_coverage`, `standard_agreement` and `certification` are readings of the paperwork, not judgements about it. If a Line block has no 'Treatment standard stated on this line' row, `standard_agreement` is 'standard_not_stated' -- that is a finding about the manifest, and it is common. NEVER infer the treatment standard from the description, from the waste code, from the container, or from what you know about how that kind of waste is usually treated.
5. `restriction_call` IS DECIDED ONLY BY THE RULEBOOK BELOW, from `notice_status`, `code_coverage` and `standard_agreement`. Work through the seven steps IN ORDER and STOP at the first that fires:
   a. notice_status 'none_cited' -> 'gap'.
   b. notice_status 'on_file_only' -> 'gap'.
   c. notice_status 'attached_superseded' -> 'gap'.
   d. code_coverage 'code_not_listed' -> 'gap'.
   e. standard_agreement 'different_standard' -> 'gap'.
   f. notice_status 'attached_current' AND code_coverage 'code_listed' AND standard_agreement 'same_standard' -> 'covered'.
   g. anything else -> 'not_determined'.
6. AN ATTACHED, CURRENT NOTICE THAT COVERS THE CODE IS NOT ON ITS OWN A CLEARANCE. A line whose block states NO treatment standard is 'not_determined'. It is not 'covered'. Nothing disagrees, and nothing has been checked either -- the manifest has not said what standard this load is going for. This is the step a confident reader skips, and on this rulebook it is the wrong answer rather than a defensible one.
7. `certification_call` IS DECIDED ONLY FROM `certification`: 'signed' -> 'certified', 'unsigned' -> 'uncertified', 'no_notice_to_read' -> 'not_determined'. It is asked of every line independently of its restriction call, and a line whose notification is covered can still be sitting on an unsigned certification.
8. THE NOTICE'S `Effective date` IS NOT READ BY THIS RULEBOOK. No call depends on it, no date defect is planted anywhere in this corpus, and a worksheet that reports one has invented it.
9. 'not_determined' IS A REAL ANSWER AND YOU ARE EXPECTED TO USE IT. THE TWO WAYS OF BEING WRONG HERE DO NOT COST THE SAME AND MUST NOT BE TRADED OFF AGAINST EACH OTHER. Calling a `gap` 'covered' is a FALSE CLEAR: a restricted load leaves with no notification in force behind it and nobody re-reads it. Calling a 'covered' line a `gap` is a FALSE ALARM: a load is held and an hour is spent proving the paperwork was fine. Do not guess in either direction; apply the rulebook.
10. Copy `line_code`, `line_label` and `notice_cited` verbatim from the pack. Use the exact allowed value for every field that lists them, and return every field for every line.

NOTIFICATION RULEBOOK (the authority for `restriction_call` and `certification_call`; this is an ILLUSTRATIVE rulebook written for this kit, and the restricted-waste and treatment-standard code schemes in it are INVENTED and reproduce no real code list)

WHAT COUNTS AS A LINE ON THE WORKSHEET
  - A line is on the worksheet if and only if it is a LINE OF THIS MANIFEST that the shipment carries.
  - A line struck from the manifest before dispatch is NOT on the worksheet, even though its code, its description AND ITS WHOLE DETAIL BLOCK are still printed in the pack.
  - A line on the generator's approved shipping schedule with nothing shipped under it on this manifest is NOT on the worksheet. A schedule is an approval, not a shipment.
  - A line carried on an EARLIER shipment under the same notice is NOT on the worksheet. It is a previous shipment, referred to here for context.

WHAT THE `Attached Notices` ROSTER IS
  Every pack prints an `Attached Notices` section naming the notice codes that ACCOMPANY this shipment. A pack also prints `Notice RN-nnnn` blocks for notices that are merely ON FILE for this generator, in the same layout, under the same kind of heading, mixed in among the attached ones. THE ROSTER IS THE ONLY THING THAT SAYS WHICH IS WHICH. A notice block being printed in the pack is not evidence that it accompanies the shipment, and this is the single commonest way to clear a line that should not be cleared.

THE INVENTED RESTRICTED WASTE CODES USED IN THIS PAPERWORK
  RC-SLV    solvent-bearing restricted stream, invented code
  RC-OIL    oil-bearing restricted stream, invented code
  RC-MTL    metal-bearing restricted stream, invented code
  RC-CYN    cyanide-bearing restricted stream, invented code
  RC-ACD    acidic restricted stream, invented code
  RC-SLG    sludge or settled-solids restricted stream, invented code

THE INVENTED TREATMENT STANDARDS USED IN THIS PAPERWORK
  TS-INCIN  destruction by incineration, invented standard
  TS-RECOV  solvent or oil recovery, invented standard
  TS-STAB   stabilisation of the solids, invented standard
  TS-NEUT   neutralisation, invented standard
  TS-DEACT  chemical deactivation, invented standard

THE CONTAINER TYPES USED IN THIS PAPERWORK
  drum_steel   a steel drum
  drum_poly    a plastic drum
  tote_poly    a plastic intermediate bulk container
  ibc_steel    a steel intermediate bulk container
  box_fibre    a fibreboard box, usually lined
  bulk_tanker  a road tanker loaded in bulk

THE FOUR STATED FACTS, each read between the manifest line and the notice it cites
  notice_status:
    attached_current       the line cites a notice, that notice code is on this pack's `Attached Notices` roster, and the notice's own `Status` row says it is current
    attached_superseded    the line cites a notice, that notice code is on the roster, but the notice's own `Status` row says it has been superseded by a later notice. A superseded notice is not the notification in force, whatever else it says
    on_file_only           the line cites a notice whose block IS printed in this pack but whose code is NOT on the `Attached Notices` roster. It is on file for this generator; it does not accompany this shipment
    none_cited             the line's block carries no `Restriction notice cited on this line` row at all. Nothing was cited, so there is no notice to read
  code_coverage:
    code_listed            the cited notice's `Codes covered` row lists this line's restricted waste code
    code_not_listed        the cited notice carries a `Codes covered` row and this line's restricted waste code is not on it
    no_notice_to_read      the line cites no notice, so there is no `Codes covered` row anywhere to check
  standard_agreement:
    same_standard          the treatment standard the line states is the one the cited notice's `Treatment standard certified` row names
    different_standard     the line states a treatment standard and the cited notice certifies a different one
    standard_not_stated    the line's block carries no `Treatment standard stated on this line` row at all. The notice still certifies one - this is a gap in the manifest, not in the notice
    no_notice_to_read      the line cites no notice, so there is nothing certified to compare against
  certification:
    signed                 the cited notice's `Certification signed` row says yes
    unsigned               the cited notice's `Certification signed` row says no
    no_notice_to_read      the line cites no notice, so there is no certification row to read

THE RESTRICTION CALL -- work through IN ORDER, stop at the first that fires
  1. NOTHING WAS CITED. If notice_status is `none_cited`, the worksheet records `gap`. A restricted line moving with no notification named against it is the plainest gap there is.
  2. THE NOTICE DOES NOT ACCOMPANY THE SHIPMENT. If notice_status is `on_file_only`, the worksheet records `gap`. The notice block is printed in the pack and everything in it may be perfectly in order; it is not on the `Attached Notices` roster, so it is not travelling with the load. THIS IS THE STEP A CONFIDENT READER SKIPS, because the notice is right there on the page.
  3. THE NOTICE IS SUPERSEDED. If notice_status is `attached_superseded`, the worksheet records `gap`. A notice the pack itself says has been replaced is not the notification in force, and the replacement is not in this pack.
  4. THE CODE IS NOT COVERED. If code_coverage is `code_not_listed`, the worksheet records `gap`. A notification that covers other codes does not cover this one.
  5. THE STANDARD DISAGREES. If standard_agreement is `different_standard`, the worksheet records `gap`. The line says one treatment standard and the notification certifies another; they cannot both describe this load.
  6. EVERYTHING LINES UP. If notice_status is `attached_current` AND code_coverage is `code_listed` AND standard_agreement is `same_standard`, the worksheet records `covered`.
  7. ANYTHING ELSE. The worksheet records `not_determined`. In practice this is one case and it is the second trap on the page: a CURRENT, ATTACHED notice that DOES cover the code, on a line that states no treatment standard at all. Nothing disagrees - and nothing has been checked either, because the manifest has not said what standard this load is going for. A line recorded as covered is a line nobody re-reads.

THE CERTIFICATION CALL
  1. If certification is `signed`, the worksheet records `certified`.
  2. If certification is `unsigned`, the worksheet records `uncertified`.
  3. If certification is `no_notice_to_read`, the worksheet records `not_determined`.
  4. THE TWO CALLS ARE INDEPENDENT AND ARE ASKED OF EVERY LINE. A line whose notification is `covered` can still be sitting on an unsigned certification, and a line with a `gap` can be sitting on a signed one. Answering one column from the other is how an uncertified load reaches a reviewer as clean.

THE EFFECTIVE DATE IS NOT READ
  Every notice block prints an `Effective date`. THIS RULEBOOK DOES NOT READ IT and no call above depends on it. Every effective date in this corpus is on or before its shipment's dispatch date, so there is no date defect planted anywhere and a worksheet that reports one has invented it. It is printed because a real notice carries one, and because a reader who starts reasoning from it is reasoning from a fact this rulebook does not use.

WHY `not_determined` IS A REAL ANSWER, AND WHY THE TWO DIRECTIONS DIFFER
  A worksheet that never says `not_determined` is not a confident worksheet, it is a guessing one. The two directions of being wrong here do not cost the same and must never be averaged. Calling a GAP `covered` is a false clear: a restricted load leaves with no notification in force behind it and nobody looks at it again. Calling a COVERED line a `gap` is a false alarm: a load is held and somebody spends an hour proving the paperwork was fine all along. The first is a safety and regulatory matter, the second is a delay, and one number reporting both tells a reviewer nothing about either.

Return these:
- shipment_id (string) -- the shipment identifier from the Shipment section, verbatim
- lines (array of objects) -- one object per MANIFESTED line, each carrying:
    - line_code (string) -- the line code from the Manifest row, verbatim (for example WL-4417). THIS IS THE KEY -- one object per MANIFESTED line and no others
    - line_label (string) -- the line's description, copied verbatim from its own Line section's Description row
    - notice_cited (string) -- the restriction notice code this line cites, copied verbatim from its Line section's 'Restriction notice cited on this line' row (for example RN-4471). If the Line block carries no such row at all, return the exact string 'none'
    - notice_status (enum) one of: attached_current, attached_superseded, on_file_only, none_cited -- the standing of the notice this line cites. 'none_cited' when the Line block has no 'Restriction notice cited on this line' row. Otherwise find the pack's 'Attached Notices' roster: if the cited code is NOT on that roster, the answer is 'on_file_only' -- the notice block is printed in this pack but does not accompany the shipment. If it IS on the roster, read that notice block's own 'Status' row: 'current' -> 'attached_current', superseded by another notice -> 'attached_superseded'. A NOTICE BEING PRINTED IN THE PACK IS NOT EVIDENCE THAT IT IS ATTACHED
    - code_coverage (enum) one of: code_listed, code_not_listed, no_notice_to_read -- compare the line's 'Restricted waste code on this line' row against the CITED notice's 'Codes covered' row. 'code_listed' when the code appears on it; 'code_not_listed' when it does not; 'no_notice_to_read' only when the line cites no notice at all. Read the cited notice's block whether or not that notice is attached -- this fact is about what the notice says, not about whether it travels
    - standard_agreement (enum) one of: same_standard, different_standard, standard_not_stated, no_notice_to_read -- compare the line's 'Treatment standard stated on this line' row against the cited notice's 'Treatment standard certified' row. 'same_standard' when the line states one and it matches; 'different_standard' when the line states one and it does not; 'standard_not_stated' when the Line block has NO 'Treatment standard stated on this line' row at all; 'no_notice_to_read' only when the line cites no notice. Never infer the standard from the description, from the waste code or from what you know about how that kind of waste is usually treated
    - certification (enum) one of: signed, unsigned, no_notice_to_read -- read the cited notice's 'Certification signed' row: yes -> 'signed', no -> 'unsigned'. 'no_notice_to_read' only when the line cites no notice
    - restriction_call (enum) one of: covered, gap, not_determined -- the worksheet's notification call for this line, decided STRICTLY by the shipped rulebook from `notice_status`, `code_coverage` and `standard_agreement` and nothing else. Work through the seven steps IN ORDER, stopping at the first that fires. (1) none_cited -> 'gap'. (2) on_file_only -> 'gap'. (3) attached_superseded -> 'gap'. (4) code_not_listed -> 'gap'. (5) different_standard -> 'gap'. (6) attached_current AND code_listed AND same_standard -> 'covered'. (7) anything else -> 'not_determined'. A NOTICE PRINTED IN THE PACK IS NOT AN ATTACHED NOTICE, and a current attached notice covering the code on a line that states NO treatment standard is 'not_determined', never 'covered'
    - certification_call (enum) one of: certified, uncertified, not_determined -- the worksheet's certification call, decided STRICTLY from `certification`: 'signed' -> 'certified', 'unsigned' -> 'uncertified', 'no_notice_to_read' -> 'not_determined'. Asked of every line independently of its restriction call -- a line whose notification is covered can still be sitting on an unsigned certification

Return a JSON object with exactly these top-level keys: shipment_id, lines
`lines` is an array. Return it empty only if the manifest carries no line at all.

SHIPMENT PACK
-------------
Shipment
--------
LDR-0002
Dispatched: 2026-11-26

SYNTHETIC RECORD. This pack was generated for a test kit. It is not a real manifest, not a
real restriction notice and not a real shipment, and no real generator, transporter,
receiving facility, site or waste stream is named in it. The restricted-waste codes and the
treatment standards used below are invented for this kit and are not any real code list.

Manifest
--------
Manifest MN-96315, dispatched 2026-11-26. Lines as manifested:

  WL-2879   Baghouse dust from metal recovery          RC-SLG  2 x bulk_tanker
  WL-8895   Spent pickling acid from the strip line    RC-SLV  1 x drum_poly
  WL-3971   Still bottoms from solvent recovery        RC-ACD  12 x ibc_steel
  WL-6417   Paint booth filters and overspray          RC-OIL  1 x box_fibre
  WL-7652   Oil-water separator sludge                 RC-MTL  8 x drum_poly

Attached Notices
----------------
The following restriction notices accompany this shipment:
  RN-3475
  RN-5897
  RN-6802
  RN-8729
  RN-9288
Any other Notice block printed in this pack is ON FILE for this generator and does NOT
accompany this shipment.

Notice RN-9783
--------------
Effective date: 2026-09-22
Status: current
Codes covered: RC-MTL, RC-ACD
Treatment standard certified: TS-NEUT
Certification signed: yes

Notice RN-3475
--------------
Effective date: 2026-02-09
Status: current
Codes covered: RC-SLG, RC-SLV
Treatment standard certified: TS-DEACT
Certification signed: no

Notice RN-6802
--------------
Effective date: 2026-06-16
Status: current
Codes covered: RC-OIL, RC-SLG, RC-SLV
Treatment standard certified: TS-DEACT
Certification signed: no

Notice RN-5897
--------------
Effective date: 2026-06-06
Status: superseded by RN-2974
Codes covered: RC-ACD, RC-OIL
Treatment standard certified: TS-DEACT
Certification signed: yes

Notice RN-8729
--------------
Effective date: 2026-10-12
Status: current
Codes covered: RC-CYN, RC-SLV, RC-SLG
Treatment standard certified: TS-DEACT
Certification signed: yes

Notice RN-9288
--------------
Effective date: 2026-03-04
Status: current
Codes covered: RC-MTL, RC-ACD, RC-SLV
Treatment standard certified: TS-STAB
Certification signed: yes

Line WL-2879
------------
Description: Baghouse dust from metal recovery
Restricted waste code on this line: RC-SLG
Restriction notice cited on this line: RN-8729
Container: bulk_tanker
Quantity: 2 x 1,000 L

Line WL-8895
------------
Description: Spent pickling acid from the strip line
Restricted waste code on this line: RC-SLV
Restriction notice cited on this line: RN-3475
Treatment standard stated on this line: TS-INCIN
Container: drum_poly
Quantity: 1 x 1,000 L

Line WL-3971
------------
Description: Still bottoms from solvent recovery
Restricted waste code on this line: RC-ACD
Restriction notice cited on this line: RN-9783
Treatment standard stated on this line: TS-NEUT
Container: ibc_steel
Quantity: 12 x 25 kg

Line WL-6417
------------
Description: Paint booth filters and overspray
Restricted waste code on this line: RC-OIL
Restriction notice cited on this line: RN-5897
Treatment standard stated on this line: TS-DEACT
Container: box_fibre
Quantity: 1 x 22,000 L

Line WL-7652
------------
Description: Oil-water separator sludge
Restricted waste code on this line: RC-MTL
Restriction notice cited on this line: RN-9288
Container: drum_poly
Quantity: 8 x 205 L

Manifest Notes
--------------
Assembled by the yard for review. Nothing in this pack has been accepted, no load
has been cleared against it and no certification has been countersigned.

Raw responseThe raw response

{"shipment_id":"LDR-0002","lines":[{"line_code":"WL-2879","line_label":"Baghouse dust from metal recovery","notice_cited":"RN-8729","notice_status":"attached_current","code_coverage":"code_listed","standard_agreement":"standard_not_stated","certification":"signed","restriction_call":"not_determined","certification_call":"certified"},{"line_code":"WL-8895","line_label":"Spent pickling acid from the strip line","notice_cited":"RN-3475","notice_status":"attached_current","code_coverage":"code_listed","standard_agreement":"different_standard","certification":"unsigned","restriction_call":"gap","certification_call":"uncertified"},{"line_code":"WL-3971","line_label":"Still bottoms from solvent recovery","notice_cited":"RN-9783","notice_status":"on_file_only","code_coverage":"code_listed","standard_agreement":"same_standard","certification":"signed","restriction_call":"gap","certification_call":"certified"},{"line_code":"WL-6417","line_label":"Paint booth filters and overspray","notice_cited":"RN-5897","notice_status":"attached_superseded","code_coverage":"code_listed","standard_agreement":"same_standard","certification":"signed","restriction_call":"gap","certification_call":"certified"},{"line_code":"WL-7652","line_label":"Oil-water separator sludge","notice_cited":"RN-9288","notice_status":"attached_current","code_coverage":"code_listed","standard_agreement":"standard_not_stated","certification":"signed","restriction_call":"not_determined","certification_call":"certified"}]}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check a hazardous waste shipment's paperwork before it's accepted — 50 shipment packs. One model answered, and every answer was then graded Five different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

No judge model anywhere. The scorer is exact match with light normalisation for cells and set comparison on line codes for identification, against a mechanically-derived gold.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend is recorded in the shared call ledger, not on this page.

Grading unitWhat the grading figure prices

evals/judge.py and evals/baseline.py are pure code and cost $0.00 to run against any result set, as many times as you like.

The gradersFive ways to grade

THE FLOOR WINS THREE COLUMNS AND THEY ARE NOT QUALITY COLUMNS. Cost: $0.00 against $0.0089438 per query. Latency: effectively zero against 13362 ms p50 and 22166 ms p95. And it needs no key, so it runs on a laptop with no network — the local app ships it as a mode, not only as an eval, and the failure screenshot on this page is the floor answering a real pack. A fourth column is a TIE and must not be read as a win: 0 false alarms of 75, the same as the run, because the floor cannot produce a false alarm at all.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Set limitationsWhat this set cannot show

The corpus is deliberately NOT balanced across the restriction call — 110 gap, 90 not_determined, 75 covered — because the two directions have different denominators by design and a 1:1:1 split would make the false-clear denominator the smallest of the three. Within gap, the five causes are dealt to fixed counts (20 citing nothing, 25 un-rostered, 25 superseded, 20 code-not-covered, 20 standard-disagrees) so no single cause can dominate the rate.

The false-clear rate on this page is a rate over 110 lines whose five causes were chosen by the same person who wrote the rulebook, in the proportions that person picked. It is not an estimate of how often a real notification queue would be cleared wrongly, and the proportions are the reason: a facility whose commonest defect is a superseded notice and one whose commonest defect is an unsigned certification would get different numbers off the same run.

Authoring any one of those is a corpus change, which re-derives every gold label and moves both headline denominators — so it is a new dataset version and a new run, not an addition to this one. None of the five was attempted. No probe set was authored for this kit.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

The free code floor (evals/baseline.py), the scorer (evals/judge.py) and the pre-spend gate (evals/check_labels.py) are all pure code and cost $0.00 to run against any result set. The figure above is the ONE scored run's own token counts (50 packs) priced at Google Gemini 3 Flash's published rate. ⚠︎ IT IS NOT WHAT BUILDING THIS KIT COST. Four further paid passes sit outside it and all four are committed in results/: a 6-pack calibration that MEASURED the output ceiling ($0.0449), the worked-example capture ($0.0060), 4 prompt-split calls at max_tokens=1 ($0.0069), and THE INJECTION PROBE — 47 calls, 108870 output tokens, $0.3266 on the OUTPUT side alone. ⚠︎ THE PROBE'S INPUT SIDE IS NOT PRICED: evals/injection.py records output tokens and not input, so its total is not stated here rather than estimated. The one live UI screenshot is one call on the same pack as the worked example and is priced at that call's own measured tokens ($0.0060), which is the nearest measurement rather than a count of its own.

Cost driversWhat actually moves the bill

• The fixed prefix — system prompt (1264 tokens), notification rulebook (1886) and field schema (959) — is 4109 of 5153 tokens on the worked example, 79.7 pct, and is byte-identical on every one of the 50 calls. The shipment pack itself is 1044 tokens. The bill is the rule and the instructions, not the paperwork.
• Provider-side reasoning. 79896 of the run's 105144 output tokens (76.0 pct) were reasoning rather than the JSON worksheet, on a reply that is a few hundred tokens of actual content. Reasoning was left at the provider's default and nothing here has measured what turning it off would cost in accuracy.
• Output length scales with the PACK, not with a fixed record — this kit's reply is a LIST of four to seven nine-field objects. The run's largest reply was 3511 output tokens against a calibration maximum of 1897 on a six-pack subset, which is why the published ceiling has the margin it has.
• THE RULEBOOK IS THE LINE ITEM THAT DOES NOT SCALE. Six invented waste codes and five treatment standards cost 1886 tokens on every call. A real facility's register of two hundred approved streams, each with its own treatment standards, would dominate every request — and at that size only the notices a manifest actually cites can be sent, which is a different kit rather than a bigger prompt.

Your volumeWhat it costs at your volume

Linear in shipment packs: each call is independent, carries the same fixed prefix and shares nothing with its neighbours, so 500 packs cost ten times 50. Nothing amortises — there is no index and no cache. ⚠︎ AND IT IS NOT LINEAR IN THE RULEBOOK, WHICH IS THE AXIS THAT ACTUALLY BREAKS: the register is sent whole in every call and its size is a property of the FACILITY, not of the shipment. Lines per pack is the second non-linear axis — the reply is a list — but at 3511 output tokens against a 32000 ceiling there is real room there.

Model choiceWhy this model

It is the tier configured in the shared .env at the repo root, and this kit did not choose it — every kit in this series inherits one connection so the setup cost does not grow with the number of kits. Nothing on this page is a claim that it is the right tier for this work; a second tier was not run.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 waste shipment packs, one completion call each, one tier. The calibration, the worked-example capture, the prompt-split calls, the injection probe and the UI screenshot are recorded separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

Eight modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pySegment

cut a pack into addressable sections on its underlined headings — Shipment, Generator Account, Manifest, Attached Notices, one Notice block per notice printed, one Line block per line, the decoy sections and the Notes. PURE CODE.

# Cut a shipment pack into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pySelect — a swap seam

map each of the nine fields to the sections that can answer it, by name and by prefix, and send the union. PURE CODE. Generator Account is mapped by nothing and therefore never leaves the machine; the decoy sections and the un-rostered Notice blocks are deliberately sent, because hiding either would be marking the model's homework.

You change it to: SECTION_HINTS maps field to section names and prefixes; NEVER_SENT names what is mapped by nothing. Adding a field or withholding a section is one dict entry, and evals/check_labels.py asserts both halves over all 50 packs.

# Pick which sections of a shipment pack are sent. Pure code -- the last deterministic step

SHIPMENT = "Shipment"
MANIFEST = "Manifest"
ROSTER = "Attached Notices"
NOTES = "Manifest Notes"
SCHEDULE = "Approved Shipping Schedule, Not Shipped On This Manifest"
EARLIER = "Earlier Shipment Under The Same Notice"
LINE_PREFIX = "Line "
NOTICE_PREFIX = "Notice "
NEVER_SENT = ("Generator Account",)

src/prompt.pyPrompt

assemble one prompt per pack: the system prompt, the whole notification rulebook RENDERED from data/rulebook.json rather than retyped, the field schema, and the selected sections.

# Assemble the notification-check prompt. ONE prompt per shipment pack, every manifested line in

SYSTEM = (
RULEBOOK_TEXT = rulebook_block()
def field_schema(shipment_fields, line_fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/adapters/__init__.pyOne call — a swap seam

one completion per pack over raw HTTP — no vendor SDK. Retries 429/5xx AND transport drops with backoff, and checks the shared daily call cap before spending.

You change it to: PROVIDER, BASE_URL, MODEL in the repo-root .env. Swapping provider or model is that file plus the same run again; two adapters ship, OpenAI-compatible and Anthropic Messages.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=600):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/rulebook.pyRulebook

the two calls, as DATA (data/rulebook.json) plus about 90 lines of lookup. One definition, three readers: the corpus generator writes gold with it, the prompt states it, and the scorer re-runs it over the model's own facts. PURE CODE.

# THE NOTIFICATION RULEBOOK, loaded from data/rulebook.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
NOTICE_STATUSES = ("attached_current", "attached_superseded", "on_file_only", "none_cited")
COVERAGES = ("code_listed", "code_not_listed", "no_notice_to_read")
STANDARDS = ("same_standard", "different_standard", "standard_not_stated", "no_notice_to_read")
CERTIFICATIONS = ("signed", "unsigned", "no_notice_to_read")
RESTRICTIONS = ("covered", "gap", "not_determined")
CERT_CALLS = ("certified", "uncertified", "not_determined")
def load(path=RULEBOOK_PATH):

src/extract.pyGuardrail

compute() — the business flag, taken afterwards over whatever came back and never over gold. PURE CODE.

# Extract one shipment pack's worksheet: segment, select, prompt, one model call, then a pure-code

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 32000
def load_fields():
def line_fields(fields=None):
def shipment_fields(fields=None):
def load_doc(shipment_id):
def documents():

evals/judge.pyJudge — a swap seam

five graders, three denominators, no model. PURE CODE, $0.00.

You change it to: five graders as separate functions with their own denominators. Adding a sixth is a function and one line in run.py; none of them calls a model.

# Score a restriction-notification run. PURE CODE -- gold is exact and every answer is one value,

def norm(v):
def code_norm(v):
def equal(got, want):
def _rows_of(rec):
def score_identification(records, golds):
CELL_FIELDS = ("line_label", "notice_cited", "notice_status", "code_coverage",
def score(fields, records, golds):
CALLS = {"restriction_call": RB.RESTRICTIONS, "certification_call": RB.CERT_CALLS}
CONFIDENT = ("covered", "gap", "certified", "uncertified")

evals/baseline.pyFree floor

the same worksheet built by regex and one shortcut, scored by the same judge. No key, no call, no spend — and it is a mode of the local app, not only an eval.

# A free, rules-and-regex worksheet builder. No model, no key, no spend -- scored by the same

def _row(block, key):
def _roster(text):
def _notices(text):
def extract_one(text, fields):
def extract(text, fields):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 5270 input and 2102 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's shipment packs are entirely synthetic (tools/build_corpus.py, seed 20260825): no real generator, transporter, receiving facility, site, shipment, manifest, notification or certification exists in the corpus, and nothing was fetched from anywhere. The six restricted-waste codes and the five treatment standards are invented for this kit and are not real code lists, and no regulation, statute, agency, standard body, published form or company procedure is named anywhere in the kit. Every pack carries a SYNTHETIC RECORD banner as the first thing in its Shipment section, so a single pack copied out of the repository still says what it is. The only outbound traffic the kit makes is one chat-completion request per pack to the configured provider, carrying the mapped sections of one pack plus the fixed prompt and the shipped rulebook — and the Generator Account section, which is the one part of a real pack a waste desk would most mind sending anywhere, is mapped by no field and never leaves the machine, measured at 0 of 50 packs. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only. The free-code-floor mode of that UI makes no outbound request at all.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser. NO API KEY IS EVER REQUESTED FROM A READER: the published pages render from committed run records and the local app renders, and answers with the free floor, with no key at all.

The experimentThe one attack that is actually shipped in the corpus

The 25 un-rostered notices are not a red-team scenario — they are 9 per cent of the corpus, dealt from a fixed count, and every one of them reads perfectly. The experiment they run is not 'can a model be talked out of an answer' but 'will a reader check the one row that makes a good-looking document irrelevant'. The free code floor is the control arm and it fails all 25; the scored run passes all 25. That is the whole kit in one comparison, and it costs nothing to reproduce: python -m evals.run --run-id b000-disposal-restrict-notice-present --baseline needs no key. The probe was fired for real on 2026-08-25 — one forced, paired injection probe was fired; see redteam: python -m evals.injection --run-id x001-disposal-restrict-injection --against r001-disposal-restrict --yes.

Each row is a boundary somebody could reasonably assume this kit has and it does not, or has and it does. Nothing here is a penetration test of a deployed service, because there is no deployed service: the kit is a corpus, a local process and one outbound completion request.

The resultThe section a real yard would most mind sending anywhere never leaves the machine — 0 of 50 packs, asserted before any spend by a gate that checks BOTH directions. And the attack this shape of paperwork actually invites is not a prompt injection at all: it is a perfectly ordinary notice printed on the page that is not travelling with the load. 25 lines in this corpus are that, the free code floor clears every one, and the scored run cleared none.

The probe forces the condition rather than counting where a seed happened to put a note, replaces the section rather than appending to it (so one variable moves, not two), and fires at the control run's own ceiling. It scores the WHOLE answer, not the verdict: all eight watched fields on every matched line are diffed against that line's own un-injected value. ⚠︎ AND BOTH MOVES WENT THE OTHER WAY. The two fields that moved were on LDR-0036/WL-1321 and LDR-0040/WL-7936 — the two lines the scored run got wrong — and both moved TOWARD the correct answer. That is indistinguishable from run-to-run variance on the hardest lines in the corpus, and no repeat pair was fired, so nothing here separates the two explanations.

HonestyWhat this does not prove

• Whether any of these boundaries hold on paperwork this kit did not generate.
• Whether the two field moves under injection are an injection effect or sampling variance. No repeat pair was fired.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_notification_query fires when the pack carries at least one line whose `restriction_call` is "gap" AND whose `certification_call` is "uncertified" — two independent failures on the same drum: there is no notification in force covering that line, and the certification on the notice it did cite is not signed either. Both values come out of the same reply; the rule is run afterwards in pure code, over whatever the model returned, and never over gold. A reply carrying no line the rule can read returns None rather than False — an unknown is not a pass.

src/extract.py::compute(), called by extract() on every pack, by the local app on every click INCLUDING in free-code-floor mode, and by evals/baseline.py on the floor's own output, so the two arms are routed by identical code. The rulebook lookup it reads lives in a SEPARATE module, src/rulebook.py, which is also what the corpus generator wrote gold with and what src/prompt.py states to the model — one definition, three readers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• IT IS NOT THE FALSE-CLEAR MEASUREMENT AND MUST NOT BE READ AS ONE. The flag is a routing signal over a whole pack; the false-clear count is a per-line comparison against gold with its own denominator. A run could score 50 of 50 on the flag and still clear gaps, because a pack with one uncertified gap trips the flag whatever the other lines say.
• It is not a compliance determination, an authorisation, an acceptance, a rejection, a clearance or a filing. It is a value in a JSON body and a red row in a local page.
• It is not any facility's acceptance procedure. No regulation, agency guidance, published form, industry standard or company procedure was consulted, and none is reproduced. A real review weighs how much is on the line, what the receiving facility is permitted to take, what the driver can tell you that the paperwork cannot, and what the same generator's last five shipments looked like.
• It is not the no-gold consistency diagnostic, which is a different thing on this page and is deliberately not called a guardrail: it re-runs the rulebook over a row's own facts and needs no labels, and on this run it caught 0 of 3 call errors — because every one of them applied the rule correctly to a misreading.

WatchedWhat is watched, and why that one

The kit measures 24 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

2 measured by the latest run 22 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

5 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

comply · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on calls_scored, documents, false_alarm_of, false_clear_of, gold_lines, lines, max_tokens, not_determined_in_gold, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 11 chips that all say so.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 2 chips that all say so.

DeviationsWhat deviated

0 breaches across 5 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-measure the firing rate whenever the corpus composition or the rulebook changes — evals/check_labels.py does it automatically on every run and fails the build outside 10-60 pct. Re-score the flag against gold whenever compute()'s condition changes. And re-read the candidate table before widening it: four of the five alternatives fire on 62-94 pct of packs, which is what a flag looks like just before it stops being read.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the rulebook is a JSON file read with json, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install — and the free code floor runs with no key at all.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing depends on anything. requirements.txt names no package, and every import in src/ and evals/ is either the standard library or another file in this kit. The dependency graph is a single node, which is the whole reason a cold clone runs the gate, the free floor and the offline UI with no install and no network.

The other sideWhat a framework costs you

• The cost of no framework is that four things are hand-rolled and each has a comment saying what it cost to get right: the retry loop (a 503 and a dropped socket are not failed documents, and the socket timeout is armed by the token ceiling), the prompt-token measurement (nested prefixes at max_tokens=1, because there is no local tokenizer), the section regex (which falls back to one whole-document segment rather than pretending), and the .env reader (merged key by key, so a kit-local file holding only MODEL keeps the shared key).
• The cost of the rulebook being DATA is that nothing type-checks it. A malformed rulebook.json is a KeyError at import time rather than a validation error with a line number. That is a real cost and it is accepted because the alternative — a schema library — is a dependency for a file a forker is expected to open and rewrite by hand.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-disposal-restrict on the fast tier, 2026-08-25. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 2 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

1 run not plotted. b000-disposal-restrict-notice-present recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineWhich lines reached the worksheet

Did the run list exactly the lines this manifest carries? Micro precision and recall over line codes, with the two directions named and never averaged: a MISSED line is a real manifested line left off, and a PHANTOM line is a shipment on the worksheet that nobody made. Phantoms are broken down by which decoy produced them.

MethodHow the test was run

evals/judge.py::score_identification, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, which records the manifested lines AND the decoy codes for every pack; evals/check_labels.py asserts before any spend that every decoy code is in the pack, collides with no manifested line, and that each of the three kinds clears a floor of 8.

How tight can the band be? No tolerance band — a code is on the worksheet or it is not, compared after whitespace stripping and upper-casing.

Cadence: Re-score on any change to data/rulebook.json's what_is_a_line_here, to src/prompt.py's rule 1, or to the corpus's decoy composition.

The decisionWhen to reach for it

In one linePer-(line, field) exact match, light normalisation

For every GOLD line and each of the 8 scored fields, does the returned value equal gold's after whitespace and case normalisation? The denominator is GOLD's lines, not the run's, so a line the run never returned scores as a miss on all 8 of its cells rather than disappearing.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, whose four stated facts are what the pack states and whose two calls are the rulebook's lookup over them.

How tight can the band be? Exact match after stripping whitespace and trailing punctuation and lower-casing. No fuzzy matching anywhere.

Cadence: Re-score on any change to data/fields.json, to the corpus layout, or to the normalisation in evals/judge.py::norm.

The decisionWhen to reach for it

In one line⚑ The headline — the two directions, never averaged

FALSE CLEAR: of the 110 lines gold calls a gap, how many did the run call covered? FALSE ALARM: of the 75 lines gold calls covered, how many did the run call a gap? Two counts, two denominators, and evals/judge.py will not produce a blended rate over all lines.

MethodHow the test was run

evals/judge.py::score_calls, the directions block, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: gold's restriction_call, which is the shipped rulebook's own lookup over the four facts each pack states.

How tight can the band be? No band. A call is covered, gap or not_determined; the directions are counted from the (gold, got) pair and nothing is weighted.

Cadence: Re-score on any change to data/rulebook.json's restriction_rule or to the corpus's joint composition table — either moves a denominator, and a rate whose denominator moved is not comparable with the one on this page.

The decisionWhen to reach for it

In one lineHow often it settles what the paperwork does not

Of the 110 calls across both columns that gold leaves not_determined, how many did the run answer with a confident value anyway (overconfident), and of the 440 it does settle, how many did the run decline (overcautious)? Two more directions, also never netted.

MethodHow the test was run

evals/judge.py::score_calls, the determinacy block, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: gold's two calls, both rulebook lookups.

How tight can the band be? A covered, gap, certified or uncertified answer counts as confident; an unanswered cell is counted separately and is never folded into either direction.

Cadence: Re-score on any change to step 6 or 7 of the restriction rule, or to the count of trap-2 lines in the corpus.

The decisionWhen to reach for it

In one lineThe business flag, against the same rule over gold

Does needs_notification_query fire on exactly the packs where the condition holds? A four-cell matrix with recall and precision. A reply that did not answer is counted as unanswered, never as a true negative.

MethodHow the test was run

evals/judge.py::score_flag, in-process, no key and no model. The positive class is the same src/extract.py::compute() run over gold's own lines.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute() run over GOLD's own lines — the identical function that runs over the reply, so the flag is measured and not the rule.

How tight can the band be? The condition is exact: at least one line whose restriction call is gap AND whose certification call is uncertified. No weighting, no score, no band.

Cadence: Re-score whenever compute()'s condition changes, and re-run evals/check_labels.py, which fails the build if the new condition leaves the 10-60 pct band.

The decisionWhen to reach for it