The business caseThe problem this solves

A waste shipment does not arrive as one document. It arrives as a manifest and a stack of approved profiles, and somebody has to lay them side by side: which lines is this load actually carrying, does each line's stream class match the profile it points at, is the physical state one that profile approved, and is the drum a container type the profile lists. Three things on the page look like evidence and are not — a matching stream code on a line that states no physical state, which says what the generator CALLED the stream and nothing about what is in the drum; a profile on file that nothing on this manifest is shipped under; and a line struck before dispatch whose whole detail block is still printed. Someone laying a shipping manifest beside the approved profile each of its lines names and working through, line by line, three comparisons the two documents may or may not settle between them: does the stream class on the line match the one the profile approved, is the physical state one the profile lists, and is the drum a container type the profile lists. It is minutes per shipment and it is every shipment, and the part that goes wrong is not the comparison: it is treating a matching stream code as a reconciliation when the line states no physical state at all, and quietly recording a settled answer where the paperwork says nothing.

The inputThe actual shipment packs

The corpus is 50 shipment packs, 0.17 MB (txt 50). Plain text, one format, invented rather than fetched — a real waste manifest names a real generator, a real transporter and a real receiving facility, carries account and site references both sides treat as commercial, and states a classification somebody signed for; there is no public corpus of (manifest, profile, correct worksheet) triples for the same reason there is no public corpus of bank statements. Generating it also makes the label mechanical: gold's two calls are the shipped rulebook's own lookup run over the same three comparisons the pack states, never somebody's reading of what the load really is.

Shipment
--------
MFP-0001

SYNTHETIC RECORD. This pack was generated for a test kit. It is not a real manifest, not a
real profile and not a real shipment, and no real generator, transporter, receiving
facility, site or waste stream is named in it. The stream-class scheme used below is
invented for this kit and is not any real code list.

Generator Account
-----------------
Brackenhill Fabrication (multi-site generator), generator account GA-73782, site reference
ST-6965, billing contact BC-5986, transporter account TA-6051.

Manifest
--------
Manifest MN-61834, dispatched 2026-02-01. Lines as manifested:

  ML-9833   Grit from an abrasive blast cabinet          WP-9951   2 x box_fibre
  ML-8636   Recovered solvent still bottoms              WP-2812   2 x ibc_steel
  ML-7017   Small-container consolidation from QC        WP-7385   5 x box_fibre
  ML-1707   Filter cake from the plate press             WP-1407   2 x tote_poly

Profile WP-9951
---------------
Approved stream class: SC-INR
Physical states approved: solid
Containers approved: box_fibre, drum_poly

Profile WP-2812
---------------
Approved stream class: SC-SOL
Physical states approved: liquid, solid
Containers approved: drum_steel, ibc_steel

Profile WP-7385
---------------
Approved stream class: SC-LAB
Physical states approved: solid
Containers approved: box_fibre, drum_steel

Profile WP-1407
---------------
Approved stream class: SC-SLG
Physical states approved: sludge, solid
Containers approved: no approved container list is recorded on this profile.

Line ML-9833
------------
Description: Grit from an abrasive blast cabinet
Profile named on this line: WP-9951
Stream class on this line: SC-OIL
Physical state on this line: solid
Container: box_fibre
Quantity: 2 x 30 kg

Line ML-8636
------------

The outcomeWhat a good result looks like

A worksheet row per manifested line — the line code, its description, the profile it names, the three comparisons the two documents state, and the two calls the shipped rulebook makes from them — plus one pure-code query flag taken from two of those values. Every call carries the rulebook's own reasoning and not_determined is a first-class answer. Nothing here authorises a shipment, accepts or rejects a load, clears a line for transport, certifies anything or files anything.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Working a week of inbound shipment paperwork before a qualified reviewer opens it — the one tier measured here — 275 of 275 on both calls, with 0 over-confident calls of 190
  100 pct on the stream call against the free code-match floor's 45.45 pct, and 0 pct over-confidence against the floor's 100 pct. It also left every one of the 39 decoy codes off, against the floor's 13 phantom shipments, and its query flag is 20 of 20 against the floor's 14 with 6 false alarms.
• Deciding whether to trust the not_determined column at all — the column, with the two causes in front of you
  The 190 open calls have two completely different causes and averaging them would hide it. 110 are open because the MANIFEST states nothing — 70 lines name the same class with no physical state, 40 state no class; 80 are open because the PROFILE lists nothing — no approved container list at all. The run reported all 190 correctly and declined none of the 360 it should have settled.

And where nothing here is good enough:

• Deciding whether a cheaper or a more deliberate model would do better here — nothing — this page cannot answer it, and it cannot answer it more completely than most
  Only one model was run, and EVERY grader on the page is at 100 pct. There is no tier comparison, no repeatability pair and no ablation, and even if all three were fired the corpus could not rank them: a grader nothing fails ranks nothing. The next honest move is a harder corpus, not another run.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-manifest-profile. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts, the 50-pack picker populates, the worksheet draws its six column headings and the shipped rulebook renders in full from data/rulebook.json — including the invented stream-class register; clicking Build worksheet returns the no-key sentence rather than a stack trace, and the query row reads “not computed — the reply carried no line the rule could read” rather than “no”. python -m evals.check_labels passes with no network access at all, and so does the free floor.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per shipment pack

Current processWhat it replaces

Someone laying a shipping manifest beside the approved profile each of its lines names and working through, line by line, three comparisons the two documents may or may not settle between them: does the stream class on the line match the one the profile approved, is the physical state one the profile lists, and is the drum a container type the profile lists. It is minutes per shipment and it is every shipment, and the part that goes wrong is not the comparison: it is treating a matching stream code as a reconciliation when the line states no physical state at all, and quietly recording a settled answer where the paperwork says nothing.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptArchitectureArchitectureApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 3 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings; a pack with none falls back to one whole-document segment

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 50 packs cut into 802 sections in about five thousandths of a second, in process, with no model and no network. 752 of those sections are sent; the 50 that are not are exactly the Generator Account sections, which carry the generator's name, what kind of site it is, a generator account number, a site reference, a billing contact and the transporter's account, and which no field asks for.

LicenceLicence

MIT — this repository's own licence. Every generator name, account reference, site reference, manifest number, line code, profile code, stream class and container type is invented; no real generator, transporter, receiving facility, site, product, brand, person or shipment is named, and no regulation, statute, agency, standard body, published form or company procedure is named either. THE SIX STREAM CLASSES (SC-AQ, SC-SOL, SC-OIL, SC-SLG, SC-LAB, SC-INR) ARE MADE UP FOR THIS KIT and are not a real code list. The reconciliation rulebook this kit ships (data/rulebook.json) is ILLUSTRATIVE and is not an authority.

Bring your ownBring your own shipment packs

REPLACE data/rulebook.json FIRST. It is this kit's own construction, its stream classes are invented, it reproduces no regulation or acceptance procedure, and everything downstream reads it: src/rulebook.py loads it, src/prompt.py renders it into every call, and tools/build_corpus.py writes gold with it. Then replace data/corpus/*.txt, write your own data/fields.json, and supply a gold row per pack carrying its manifested lines AND its decoys — the decoy list is what makes the phantom-line grader possible. SECTION_HINTS in src/select.py maps fields to section headings and will need editing for a different pack layout; when it does not match, selection falls back to the whole document — slower, more expensive, always correct. compute() in src/extract.py is this kit's own query rule and is the second thing to replace.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 3,984 tokens assembled, 2,295 are contexts — 58% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on MFP-0050 — the same pack the worked example above was captured on, so the four parts sum to exactly the 3984 input tokens that call actually reported. Four calls at max_tokens=1, each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. kits/UC0060-manifest-profile/results/tokens-p001-manifest-profile.json. ⛑ THE FIXED PREFIX IS 2839 of 3984 TOKENS (71.3 pct) and is paid on every one of the 50 calls.

Full promptThe prompt in full

You read a WASTE SHIPMENT PACK -- a manifest plus the approved profile each of its lines names -- and return a REVIEWER'S WORKSHEET of how the two documents compare, line by line. You return JSON and nothing else.

You are preparing a worksheet for a qualified reviewer. You never authorise a shipment, never accept or reject a load, never clear a line for transport, never certify anything, never amend a profile and never file anything with anybody. Your job is to say what the two documents state, apply the rulebook given below, and NAME THE LINES THE PAPERWORK DOES NOT RECONCILE.

RULES, in order of importance:
1. ONE OBJECT PER MANIFESTED LINE OF THIS SHIPMENT, AND NO OTHERS. A line struck from the manifest before dispatch is not manifested, even though its code, its description and its whole Line block are still printed in the pack. A profile that is on file with nothing shipped under it is not a line -- a profile is an approval, not a shipment. A line carried on an EARLIER manifest under the same profile is not a line either. Putting any of the three on the worksheet puts a shipment in front of a reviewer that nobody made.
2. EVERY COMPARISON IS AGAINST THE PROFILE THAT LINE NAMES, and no other profile in the pack. Read the line's 'Profile named on this line' row first, find that Profile block, and compare against it.
3. REPORT WHAT THE PACK STATES. `class_agreement`, `state_agreement` and `container_agreement` are readings of two documents, not judgements about them. If a Line block has no 'Physical state on this line' row, `state_agreement` is 'state_not_stated' -- that is a finding about the manifest, and it is common. NEVER infer the physical state from the description, from the container, or from what you know about how that kind of waste is usually handled.
4. `stream_call` IS DECIDED ONLY BY THE RULEBOOK BELOW, from `class_agreement` and `state_agreement`. Work through the four steps IN ORDER and STOP at the first that fires:
   a. class_agreement 'different_class' -> 'conflicts'.
   b. state_agreement 'outside_profile' -> 'conflicts'. This holds even when the classes match.
   c. class_agreement 'same_class' AND state_agreement 'within_profile' -> 'reconciles'.
   d. anything else -> 'not_determined'.
5. A MATCHING STREAM CLASS IS NOT ON ITS OWN A RECONCILIATION. A line naming the same class as its profile whose block states no physical state is 'not_determined'. It is not 'reconciles'. The class is what the generator CALLED the stream; it is not a statement about what is in the drum. This is the step a confident reader skips, and on this rulebook it is the wrong answer rather than a defensible one.
6. `container_call` IS DECIDED ONLY FROM `container_agreement`: 'listed_container' -> 'on_profile', 'unlisted_container' -> 'off_profile', 'profile_lists_none' -> 'not_determined'. It is asked of every line independently of its stream call, and a line that reconciles can still be in a container the profile does not list.
7. 'not_determined' IS A REAL ANSWER AND YOU ARE EXPECTED TO USE IT. A worksheet that never reaches for it is guessing, and a line recorded as reconciled is a line nobody re-reads.
8. Copy `line_code`, `line_label` and `profile_code` verbatim from the pack. Use the exact allowed value for every field that lists them, and return every field for every line.

RECONCILIATION RULEBOOK (the authority for `stream_call` and `container_call`; this is an ILLUSTRATIVE rulebook written for this kit, and the stream-class scheme in it is INVENTED and reproduces no real code list)

WHAT COUNTS AS A LINE ON THE WORKSHEET
  - A line is on the worksheet if and only if it is a LINE OF THIS MANIFEST that the shipment carries.
  - A line struck from the manifest before dispatch is NOT on the worksheet, even though its code, its description AND ITS WHOLE DETAIL BLOCK are still printed in the pack.
  - A profile that is on file for this generator with nothing shipped under it on this manifest is NOT on the worksheet. A profile is an approval, not a shipment.
  - A line carried on an EARLIER manifest under the same profile is NOT on the worksheet. It is a previous shipment, referred to here for context.

THE INVENTED STREAM CLASSES USED IN THIS PAPERWORK
  SC-AQ    aqueous stream, invented class - water-based, no separate organic phase recorded
  SC-SOL   solvent-bearing stream, invented class
  SC-OIL   oil-bearing stream, invented class
  SC-SLG   sludge or settled-solids stream, invented class
  SC-LAB   laboratory-pack stream, invented class - many small containers consolidated
  SC-INR   inert stream, invented class - carried on a profile but recorded as unreactive

THE CONTAINER TYPES USED IN THIS PAPERWORK
  drum_steel   a steel drum
  drum_poly    a plastic drum
  tote_poly    a plastic intermediate bulk container
  ibc_steel    a steel intermediate bulk container
  box_fibre    a fibreboard box, usually lined
  bulk_tanker  a road tanker loaded in bulk

THE THREE STATED FACTS, each a comparison between the manifest line and the profile it names
  class_agreement:
    same_class           the manifest line states a stream class and it is the same class the profile it names states
    different_class      the manifest line states a stream class and it is a different class from the one the profile it names states
    class_not_stated     the manifest line states no stream class at all. The profile still states one - this is a gap in the manifest, not in the profile
  state_agreement:
    within_profile       the physical state the manifest line states is one of the states the named profile lists as approved
    outside_profile      the physical state the manifest line states is NOT one the named profile lists
    state_not_stated     the manifest line states no physical state at all
  container_agreement:
    listed_container     the container type the manifest line states appears on the named profile's approved container list
    unlisted_container   the container type the manifest line states does NOT appear on that list, and the profile does carry one
    profile_lists_none   the named profile records no approved container list at all, so there is nothing to check the container against

THE STREAM CALL -- work through IN ORDER, stop at the first that fires
  1. THE CLASSES DISAGREE. If class_agreement is `different_class`, the worksheet records `conflicts`. Two pieces of paperwork name two different streams for the same drum, and nothing further needs checking to say so.
  2. THE STATE IS OUTSIDE THE PROFILE. If state_agreement is `outside_profile`, the worksheet records `conflicts`. A physical state the profile does not list is a disagreement even where the class matches - a profile approved for liquid says nothing about a sludge.
  3. BOTH AGREE. If class_agreement is `same_class` AND state_agreement is `within_profile`, the worksheet records `reconciles`. Both stated facts line up with the profile the line names.
  4. ANYTHING ELSE. The worksheet records `not_determined`. A MATCHING STREAM CLASS IS NOT ON ITS OWN A RECONCILIATION - the class is what the generator called the stream, and it says nothing about whether what is actually in the drum is what the profile approved. A line that names the same class as its profile and states no physical state does not reconcile; it is unchecked. This is the branch a confident reader skips.

THE CONTAINER CALL
  1. If container_agreement is `listed_container`, the worksheet records `on_profile`.
  2. If container_agreement is `unlisted_container`, the worksheet records `off_profile`.
  3. If container_agreement is `profile_lists_none`, the worksheet records `not_determined`. The container question is asked of every line independently of the stream call - a line whose classes disagree is still either in a container the profile lists, one it does not, or one there is no list for.

WHY `not_determined` IS A REAL ANSWER
  A worksheet that never says `not_determined` is not a confident worksheet, it is a guessing one. The whole value of this shape is that a reviewer can see, in one column, which lines the two documents actually settle between them and which ones somebody has to go back to the generator about. Over-confidence here costs more than caution: a line recorded as reconciled is a line nobody re-reads, and on this kind of paperwork the line nobody re-reads is the one that arrives in the wrong container.

Return these:
- shipment_id (string) -- the shipment identifier from the Shipment section, verbatim
- lines (array of objects) -- one object per MANIFESTED line, each carrying:
    - line_code (string) -- the line code from the Manifest row, verbatim (for example ML-4417). THIS IS THE KEY -- one object per MANIFESTED line and no others
    - line_label (string) -- the line's description, copied verbatim from its own Line section's Description line
    - profile_code (string) -- the profile code this line names, copied verbatim from its Line section's 'Profile named on this line' row (for example WP-2081). Every comparison below is against THAT profile's block and no other
    - class_agreement (enum) one of: same_class, different_class, class_not_stated -- compare the line's 'Stream class on this line' row against the named profile's 'Approved stream class' row. 'same_class' when the line states a class and it is the same one; 'different_class' when the line states a class and it is a different one; 'class_not_stated' when the line has NO 'Stream class on this line' row at all. Report what the paperwork says -- a missing row is a finding about the manifest, not a gap to fill in from the description
    - state_agreement (enum) one of: within_profile, outside_profile, state_not_stated -- compare the line's 'Physical state on this line' row against the named profile's 'Physical states approved' list. 'within_profile' when the stated state is on that list; 'outside_profile' when it is not; 'state_not_stated' when the line has NO 'Physical state on this line' row at all. Never infer the state from the description -- a line describing a sludge and stating no state is 'state_not_stated'
    - container_agreement (enum) one of: listed_container, unlisted_container, profile_lists_none -- compare the line's 'Container' row against the named profile's 'Containers approved' row. 'listed_container' when the container type is on that list; 'unlisted_container' when the profile carries a list and this type is not on it; 'profile_lists_none' when the profile's row says no approved container list is recorded
    - stream_call (enum) one of: reconciles, conflicts, not_determined -- the worksheet's stream call for this line, decided STRICTLY by the shipped rulebook from `class_agreement` and `state_agreement` and nothing else. Work through the four steps IN ORDER, stopping at the first that fires. (1) class_agreement 'different_class' -> 'conflicts'. (2) state_agreement 'outside_profile' -> 'conflicts'. (3) class_agreement 'same_class' AND state_agreement 'within_profile' -> 'reconciles'. (4) anything else -> 'not_determined'. A MATCHING STREAM CLASS IS NOT ON ITS OWN A RECONCILIATION: a line naming the same class as its profile and stating no physical state is 'not_determined', never 'reconciles'
    - container_call (enum) one of: on_profile, off_profile, not_determined -- the worksheet's container call, decided STRICTLY from `container_agreement`: 'listed_container' -> 'on_profile', 'unlisted_container' -> 'off_profile', 'profile_lists_none' -> 'not_determined'. Asked of every line independently of its stream call -- a line whose classes disagree is still either in a container the profile lists, one it does not, or one there is no list for

Return a JSON object with exactly these top-level keys: shipment_id, lines
`lines` is an array. Return it empty only if the manifest carries no line at all.

SHIPMENT PACK
-------------
Shipment
--------
MFP-0050

SYNTHETIC RECORD. This pack was generated for a test kit. It is not a real manifest, not a
real profile and not a real shipment, and no real generator, transporter, receiving
facility, site or waste stream is named in it. The stream-class scheme used below is
invented for this kit and is not any real code list.

Manifest
--------
Manifest MN-54946, dispatched 2026-01-04. Lines as manifested:

  ML-6134   Spent aqueous alkaline cleaner               WP-1216   1 x bulk_tanker
  ML-8132   Consolidated laboratory pack, bottles        WP-1846   7 x drum_poly
  ML-3221   Spray booth sludge and overspray             WP-8577   11 x box_fibre
  ML-2767   Spent adsorbent media, recorded inert        WP-4640   3 x tote_poly
  ML-6803   Tank-bottom oil sludge                       WP-6458   3 x drum_poly
  ML-6378   Solvent-bearing wipes and rags               WP-5135   2 x tote_poly
  ML-5321   Settled sludge from a holding tank           WP-2949   VOID before dispatch

Profile WP-1216
---------------
Approved stream class: SC-AQ
Physical states approved: liquid
Containers approved: drum_poly, tote_poly

Profile WP-1846
---------------
Approved stream class: SC-LAB
Physical states approved: solid
Containers approved: no approved container list is recorded on this profile.

Profile WP-8577
---------------
Approved stream class: SC-SLG
Physical states approved: sludge, solid
Containers approved: drum_steel, box_fibre

Profile WP-4640
---------------
Approved stream class: SC-INR
Physical states approved: solid
Containers approved: no approved container list is recorded on this profile.

Profile WP-6458
---------------
Approved stream class: SC-OIL
Physical states approved: liquid, sludge
Containers approved: drum_steel, bulk_tanker

Profile WP-5135
---------------
Approved stream class: SC-SOL
Physical states approved: liquid, solid
Containers approved: drum_steel, ibc_steel

Line ML-6134
------------
Description: Spent aqueous alkaline cleaner
Profile named on this line: WP-1216
Stream class on this line: SC-AQ
Physical state on this line: liquid
Container: bulk_tanker
Quantity: 1 x 22,000 L

Line ML-8132
------------
Description: Consolidated laboratory pack, bottles
Profile named on this line: WP-1846
Stream class on this line: SC-LAB
Container: drum_poly
Quantity: 7 x 205 L

Line ML-3221
------------
Description: Spray booth sludge and overspray
Profile named on this line: WP-8577
Stream class on this line: SC-SLG
Container: box_fibre
Quantity: 11 x 30 kg

Line ML-2767
------------
Description: Spent adsorbent media, recorded inert
Profile named on this line: WP-4640
Stream class on this line: SC-INR
Physical state on this line: sludge
Container: tote_poly
Quantity: 3 x 1,000 L

Line ML-6803
------------
Description: Tank-bottom oil sludge
Profile named on this line: WP-6458
Physical state on this line: sludge
Container: drum_poly
Quantity: 3 x 205 L

Line ML-6378
------------
Description: Solvent-bearing wipes and rags
Profile named on this line: WP-5135
Stream class on this line: SC-SOL
Physical state on this line: sludge
Container: tote_poly
Quantity: 2 x 1,000 L

Profile WP-2949
---------------
Approved stream class: SC-SLG
Physical states approved: sludge, solid
Containers approved: drum_steel, box_fibre

Line ML-5321
------------
Description: Settled sludge from a holding tank
Profile named on this line: WP-2949
Stream class on this line: SC-SLG
Physical state on this line: sludge
Container: drum_steel
Quantity: 5 x 205 L
This block was written before the line was struck and is retained in the pack for the audit
trail.

Earlier Manifest Under The Same Profile
---------------------------------------
Line ML-3978, Interceptor draw, oil and water, was carried on manifest MN-37332 under
profile WP-4640. It is not shipped again here and is listed for context only.

Manifest Notes
--------------
This pack is a draft for review.
Line ML-5321 was struck from the manifest before dispatch and is not carried on this
shipment.
Draft pack for the review queue. No acceptance, rejection or filing follows from anything
recorded here.

Raw responseThe raw response

{
  "shipment_id": "MFP-0050",
  "lines": [
    {
      "line_code": "ML-6134",
      "line_label": "Spent aqueous alkaline cleaner",
      "profile_code": "WP-1216",
      "class_agreement": "same_class",
      "state_agreement": "within_profile",
      "container_agreement": "unlisted_container",
      "stream_call": "reconciles",
      "container_call": "off_profile"
    },
    {
      "line_code": "ML-8132",
      "line_label": "Consolidated laboratory pack, bottles",
      "profile_code": "WP-1846",
      "class_agreement": "same_class",
      "state_agreement": "state_not_stated",
      "container_agreement": "profile_lists_none",
      "stream_call": "not_determined",
      "container_call": "not_determined"
    },
    {
      "line_code": "ML-3221",
      "line_label": "Spray booth sludge and overspray",
      "profile_code": "WP-8577",
      "class_agreement": "same_class",
      "state_agreement": "state_not_stated",
      "container_agreement": "listed_container",
      "stream_call": "not_determined",
      "container_call": "on_profile"
    },
    {
      "line_code": "ML-2767",
      "line_label": "Spent adsorbent media, recorded inert",
      "profile_code": "WP-4640",
      "class_agreement": "same_class",
      "state_agreement": "outside_profile",
      "container_agreement": "profile_lists_none",
      "stream_call": "conflicts",
      "container_call": "not_determined"
    },
    {
      "line_code": "ML-6803",
      "line_label": "Tank-bottom oil sludge",
      "profile_code": "WP-6458",
      "class_agreement": "class_not_stated",
      "state_agreement": "within_profile",
      "container_agreement": "unlisted_container",
      "stream_call": "not_determined",
      "container_call": "off_profile"
    },
    {
      "line_code": "ML-6378",
      "line_label": "Solvent-bearing wipes and rags",
      "profile_code": "WP-5135",
      "class_agreement": "same_class",
      "state_agreement": "outside_profile",
      "container_agreement": "unlisted_container",
      "stream_call": "conflicts",
      "container_call": "off_profile"
    }
  ]
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch mismatches between a waste shipment and its profile — 50 shipment packs. One model answered, and every answer was then graded Five different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersFive ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The code-match floor (evals/baseline.py) and the scorer (evals/judge.py) are both pure code and cost $0.00 to run against any result set. The figure above is the ONE scored run's own token counts (50 packs) priced at Google Gemini 3 Flash's published rate — the same basis cost_per_query_usd uses. ⚠︎ IT IS NOT WHAT BUILDING THIS KIT COST. Four further paid passes sit outside it and three of them are committed in results/: a 6-pack calibration that MEASURED max_tokens ($0.0366), the worked-example capture ($0.0083) and 4 prompt-split calls at max_tokens=1 ($0.0049). The fourth is the one live UI screenshot, whose tokens the screenshot script does not record — it is one call on the same pack as the worked example and is priced here at that call's own measured tokens ($0.0083), which is the nearest measurement rather than a count of its own. All 62 live calls together project to $0.3869 on this card.

Cost driversWhat actually moves the bill

• The fixed prefix — system prompt (891 tokens), reconciliation rulebook (1,150) and field schema (798) — is 2,839 of 3,984 tokens on the example call, 71.3 pct, and is byte-identical on every one of the 50 calls. The shipment pack itself is 1,145 tokens. The bill is the rulebook and the instructions, not the paperwork.
• Provider-side reasoning. 53,646 of the run's 78,042 output tokens (68.7 pct) were reasoning rather than the JSON worksheet, on a reply that is about 500 tokens of actual content. Reasoning was left at the provider's default and nothing here has measured what turning it off would cost in accuracy.
• Output length scales with the PACK, not with a fixed record — this kit's reply is a LIST. A four-line pack and a seven-line pack are not the same call, and the run's largest reply was 2,580 output tokens against a calibration maximum of 1,788 on a six-line subset.
• THE RULEBOOK IS THE LINE ITEM THAT DOES NOT SCALE. Six invented stream classes and six container types cost 1,150 tokens on every call. A real facility's register of two hundred approved streams, each with its own approved states and containers, would dominate every request — and at that size only the profiles a manifest actually names can be sent, which is a different kit rather than a bigger prompt.

Your volumeWhat it costs at your volume

Linear in shipment packs: each call is independent, carries the same fixed prefix and shares nothing with its neighbours, so 500 packs cost ten times 50. Nothing amortises — there is no index and no cache. ⚠︎ AND IT IS NOT LINEAR IN THE RULEBOOK, WHICH IS THE AXIS THAT ACTUALLY BREAKS. The register is sent whole in every call and its size is a property of the FACILITY, not of the shipment: ten times the profiles is ten times the prefix on every call, whether or not the manifest names any of them. Lines per pack is the second non-linear axis — the reply is a list — but at 2,580 output tokens against a 12,000 ceiling there is real room there.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 waste shipment packs, one completion call each, one tier. The calibration, the worked-example capture, the prompt-split calls and the UI screenshot are recorded separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the shipment pack into addressable sections, pure code — one per manifested line, one per profile that a line names, plus the manifest, the notes and any decoy section. On this kit the sections ARE the pairing: every reconciliation is between one Line block and the one Profile block it names

# Cut a shipment pack into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections are sent, pure code — section names are matched by PREFIX because every pack's Line and Profile sections are named after its own codes. The three decoy sections are deliberately SENT; the Generator Account section is mapped by nothing and never leaves the machine

You change it to: map fields to your own pack's headings, including the per-line and per-profile prefixes; unmatched falls back to the whole document. NEVER_SENT is the list of sections no field asks for and which therefore never leave the machine

# Pick which sections of a shipment pack are sent. Pure code -- the last deterministic step

SHIPMENT = "Shipment"
MANIFEST = "Manifest"
NOTES = "Manifest Notes"
ON_FILE = "Profiles On File, Not Manifested"
EARLIER = "Earlier Manifest Under The Same Profile"
LINE_PREFIX = "Line "
PROFILE_PREFIX = "Profile "
NEVER_SENT = ("Generator Account",)
SECTION_HINTS = {

src/rulebook.pyrulebook

the two worksheet calls, loaded from data/rulebook.json — a four-step stream rule with a stopping order and a three-way container rule. Data, not code, so a forker can open it and disagree with it

# THE RECONCILIATION RULEBOOK, loaded from data/rulebook.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
CLASS_AGREEMENTS = ("same_class", "different_class", "class_not_stated")
STATE_AGREEMENTS = ("within_profile", "outside_profile", "state_not_stated")
CONTAINER_AGREEMENTS = ("listed_container", "unlisted_container", "profile_lists_none")
STREAMS = ("reconciles", "conflicts", "not_determined")
CONTAINERS = ("on_profile", "off_profile", "not_determined")
def load(path=RULEBOOK_PATH):
R = load()

src/prompt.pyprompt

assemble one call per pack for every manifested line, with the rulebook RENDERED into it from data/rulebook.json rather than restated in prose — a rule the model has never been shown is not a rule

# Assemble the reconciliation prompt. ONE prompt per shipment pack, every manifested line in the

SYSTEM = (
RULEBOOK_TEXT = rulebook_block()
def field_schema(shipment_fields, line_fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key — plus the pure-code business-condition check downstream: a pack carrying a line that CONFLICTS and sits in a container the profile does not list goes back to the generator as a query

You change it to: the query rule itself — this kit ships two values on one line (the stream call conflicts and the container is off the profile) and a real desk weighs quantity, what the receiving facility is permitted to take, and what the driver can tell you. It is deliberately NOT the same function as the rulebook lookup, so changing WHAT GETS QUERIED does not change WHAT THE RULEBOOK SAYS

# Extract one shipment pack's worksheet: segment, select, prompt, one model call, then a pure-code

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 12000
def load_fields():
def line_fields(fields=None):
def shipment_fields(fields=None):
def load_doc(shipment_id):
def documents():

evals/judge.pyjudge

score identification, the extracted cells, the two calls and the determinacy split, all separately, pure code — plus a no-gold consistency diagnostic

# Score a manifest-versus-profile run. PURE CODE -- gold is exact and every answer is one value,

def norm(v):
def code_norm(v):
def equal(got, want):
def _rows_of(rec):
def score_identification(records, golds):
CELL_FIELDS = ("line_label", "profile_code", "class_agreement", "state_agreement",
def score(fields, records, golds):
CALLS = {"stream_call": RB.STREAMS, "container_call": RB.CONTAINERS}
CONFIDENT = ("reconciles", "conflicts", "on_profile", "off_profile")

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3789 input and 1560 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's shipment packs are entirely synthetic (tools/build_corpus.py, seed 20260822): no real generator, transporter, receiving facility, site, shipment, manifest or profile exists in the corpus, and nothing was fetched from anywhere. The six stream classes are invented for this kit and are not a real code list, and no regulation, statute, agency, standard body, published form or company procedure is named anywhere in the kit. Every pack carries a SYNTHETIC RECORD banner as the first thing in its Shipment section, so a single pack copied out of the repository still says what it is. The only outbound traffic the kit makes is one chat-completion request per pack to the configured provider, carrying the mapped sections of one pack plus the fixed prompt and the shipped rulebook — and the Generator Account section, which is the one part of a real pack a waste desk would most mind sending anywhere, is mapped by no field and never leaves the machine. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser.

The experimentWe did not attack it — and three of four boundaries hold

Three boundaries hold. Two were confirmed by reading the code: no code path authorises, accepts, rejects, clears or files anything, and the rulebook lookup and the query rule read only enums, so no description or note text can reach either. The third is confirmed by a CHECK RATHER THAN BY READING — evals/check_labels.py asserts, before any run may spend, that the Generator Account section is never sent and that every Line block, Profile block and decoy section is. The fourth is open. An indirect prompt injection needs a field an outside party authored, and a manifest description is exactly that. Confirmed by reading the code and by a pre-flight assertion, not by an attack run, on 2026-08-22 — no attack run was fired; see redteam.why.

The first three boundaries hold, the third confirmed by an assertion that runs before every scored run rather than by reading the code alone. The fourth is open and is named as open — a manifest is a document the generator filled in, which makes an instruction hidden in a description or a note exactly the attack this kit's shape invites, and nothing here has tried it.

The result0 attack trials, and three of four boundaries checked here hold on evidence — one of them by an assertion that runs before every scored run, not by reading the code. The one that does not is the one this kit's shape invites: a manifest is a document the generator filled in, and whether an instruction hidden in a description or a note could move a call is unmeasured.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted decoys measure whether a model reads the MANIFEST TABLE to decide what is carried when the detail blocks point the other way, and whether it declines a call the paperwork does not settle. Neither measures whether it obeys an instruction hidden in the same pack. Those are different failures and only the first two are measured here.

HonestyWhat this does not prove

• Whether an indirect prompt injection in a line description or a Manifest Notes line could move a call. No attack run exists.
• Whether the kit behaves safely against a hostile provider — a response body crafted to break the list parsing in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed document, and nothing beyond that was tested.
• Whether a real shipment pack would carry anything sensitive this kit mishandles beyond the Generator Account section. The corpus has no personal data by construction; a real pack carries named signatories, a driver, vehicle details and commercial references, and none of that path is exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_generator_query fires when the pack carries at least one line whose `stream_call` is "conflicts" AND whose `container_call` is "off_profile" — two independent disagreements about the same drum. Both values come out of the same reply; the rule is run afterwards in pure code, over whatever the model returned, and never over gold. A reply carrying no line the rule can read returns None rather than False — an unknown is not a pass.

src/extract.py::compute(), called by extract() on every pack, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by identical code. The rulebook lookup it reads lives in a SEPARATE module, src/rulebook.py, which is also what the corpus generator wrote gold with and what src/prompt.py states to the model — one definition, three readers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a compliance determination, an authorisation, an acceptance, a rejection, a clearance, a certification or a filing. Nothing here authorises a shipment, accepts or rejects a load, clears a line for transport, certifies anything, amends a profile, notifies anybody or posts anything. stream_call and container_call are proposals with the rulebook's own reasoning attached, and a qualified reviewer at the receiving facility decides.
• It is NOT a check on whether the extracted comparisons are right. If the model misreads a Line block and then applies the rulebook correctly to the misreading, the shipment is queried — or not queried — on a wrong fact, and this rule cannot tell. The consistency diagnostic in evals/judge.py is the closest thing to that check, and on this run it had nothing to catch, which is not evidence that it would have.
• It is NOT any facility's acceptance procedure. Two disagreements about one drum is this kit's own simplification, invented for this corpus. No regulation, agency guidance, published form, industry standard or company waste-acceptance procedure was consulted, and none is reproduced. A real review weighs quantity, what the receiving facility is permitted to take, what the driver can tell you that the paperwork cannot, and what the same generator's last five shipments looked like.
• It is NOT a judgement about the rulebook. The flag reads the calls the rulebook produced; if the rulebook is wrong, the flag is confidently wrong with it. The shipped rulebook is illustrative, its six stream classes are invented, and nobody qualified has reviewed it.

WatchedWhat is watched, and why that one

The kit measures 15 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 8 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 50, extraction_cells 1975, failures 0, lines 275, refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

extraction · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on documents, extraction_cells, lines, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

extraction · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 7 chips that all say so.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/extract.py, on any change to data/rulebook.json (which moves the calls the flag reads), on any change to data/fields.json's allowed values for the two calls, and on any corpus regeneration. Changing WHAT GOES BACK TO THE GENERATOR is a policy change and must be re-scored, even though it never changes what the rulebook says.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the rulebook is a JSON file read with json, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per pack — segment, select, prompt, one call, parse, compute — with no branch, no loop and no agent. The only structural difference from a flat-record sibling is that the REPLY is a list, which changes the scorer and not the pipeline.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain — the JSON extraction from a fenced reply, the list parsing, the backoff policy, the section regex, the prefix matcher and the .env reader are all code somebody has to own. The corpus generator paid its own instalment of that during the build: its first version drew each line's physical state from the class's approved list independently of the description and shipped Solvent-bearing wipes and rags ... Physical state on this line: liquid, which no assertion in the file could see and the first human to read a pack caught immediately.
• No framework means no framework's ecosystem — no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. The reply is a LIST and nothing validates its shape except src/prompt.py::parse, which fails closed to {} rather than to an empty worksheet, because an empty lines list is a legitimate answer here.
• The provider abstraction covers exactly two wire formats. A third provider is one function and one dict entry, and until somebody writes it the kit runs on two shapes.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-manifest-profile on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b000-manifest-profile-codematch, t000-manifest-profile-stub recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — blocking is computed in memory each run and is the only reduction step.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Every generator name, account reference, site reference, manifest number, line code, profile code, stream class and container type is invented; no real generator, transporter, receiving facility, site, product, brand, person or shipment is named, and no regulation, statute, agency, standard body, published form or company procedure is named either. THE SIX STREAM CLASSES (SC-AQ, SC-SOL, SC-OIL, SC-SLG, SC-LAB, SC-INR) ARE MADE UP FOR THIS KIT and are not a real code list. The reconciliation rulebook this kit ships (data/rulebook.json) is ILLUSTRATIVE and is not an authority. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineWhich lines reached the worksheet

Did the run list exactly the lines this manifest carries? Micro precision and recall over line codes, with the two directions named and never averaged: a MISSED line is a real manifested line left off, and a PHANTOM line is a shipment on the worksheet that nobody made. Phantoms are broken down by which decoy produced them — a line struck before dispatch, a profile on file with nothing shipped under it, a line from an earlier manifest, or a code that appears nowhere in the pack.

MethodHow the test was run

evals/judge.py::score_identification, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, which records the manifested lines AND the decoy codes for every pack; evals/check_labels.py asserts before any run may spend that every decoy code is in the pack, collides with no manifested line, and that each of the three kinds clears a floor of 8.

How tight can the band be? No tolerance band — a code is on the worksheet or it is not, compared after whitespace stripping and upper-casing.

Cadence: Re-score on any change to data/rulebook.json's what_is_a_line_here, to src/prompt.py's rule 1, or to the corpus's decoy composition.

The decisionWhen to reach for it

In one linePer-(line, field) exact match, light normalisation

Does the model's value for each of the seven per-line fields match gold, after trimming whitespace and punctuation, plus one shipment id per pack? THE DENOMINATOR IS GOLD'S LINES, NOT THE RUN'S: a gold line the run never returned scores as a miss on all seven of its cells rather than disappearing, so a run cannot raise this number by returning less.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every run, and the same logic the free floor is scored by — a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same values the pack states; evals/check_labels.py asserts every field is populated and in its allowed set on every one of the 275 lines, and that every value is stated in the pack it labels, before any run is allowed to spend.

How tight can the band be? There is no tolerance band — it is exact match after trimming whitespace and punctuation, never a continuous score to round.

Cadence: Re-score on any change to data/rulebook.json, src/prompt.py, data/fields.json or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe two rulebook calls, three-way each

Did the run land on the same calls the shipped rulebook produces — reconciles / conflicts / not_determined for the stream, and on_profile / off_profile / not_determined for the container? SCORED ON MATCHED LINES ONLY, and saying so is half of what makes the number honest: a call about a line the run never found has already been counted, once, as an identification failure, and counting it again here would let one mistake sink two graders.

MethodHow the test was run

evals/judge.py::score_calls, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's calls are re-derived inside the grader by the same lookup the kit publishes — src/rulebook.py, reading data/rulebook.json — so the truth this grades against can never be a separately-typed label that drifted from the rule.

How tight can the band be? No threshold — each call is one of three allowed values, and a reply returning none of them is counted as unanswered rather than folded into a correct cell.

Cadence: Re-run whenever data/rulebook.json or src/rulebook.py changes, whenever the corpus is regenerated, and on any provider or model change.

The decisionWhen to reach for it

In one line⚑ The headline — how often it says the paperwork does not settle it

Of every call this corpus's paperwork does NOT settle, how many did the run settle anyway? That is overconfident, and on a reviewer's worksheet it is the expensive direction: a line recorded as reconciled is a line nobody re-reads, so an over-confident worksheet quietly removes the reviewer it exists to serve. Its mirror, overcautious — declining a call the paperwork does settle — is counted and published beside it, never netted against it, because the two costs are different: one hides work, the other creates it.

MethodHow the test was run

evals/judge.py::score_calls — the determinacy block, in-process, no key, no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The same rulebook lookup as the grader above, with the rows partitioned by whether GOLD's answer is not_determined. Both directions come out of one pass over the same 550 calls.

How tight can the band be? A rate over a fixed denominator: 190 open calls, 360 settled ones. A run over a different corpus has a different denominator and the two rates are not comparable.

Cadence: Re-run on any change to the rulebook's not_determined branches, to the corpus's silent-line composition, or to the provider or model.

The decisionWhen to reach for it

In one lineneeds_generator_query against the same rule run over gold

Does the pure-code query decision — a line whose stream call CONFLICTS and whose container is OFF the profile — land on the same shipments it would land on if every line had been read perfectly? It is a BUSINESS CONDITION, so unlike a self-consistency check it genuinely needs labels, and saying so is half of what makes the number believable.

MethodHow the test was run

evals/judge.py::score_flag, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute(), the same function the run uses, applied to GOLD's own lines. One rule, one input shape, so a change to the rule moves both sides together and the grader cannot silently grade an old policy.

How tight can the band be? No threshold — two values per line and one comparison.

Cadence: Re-run whenever compute() changes. Changing WHAT GOES BACK TO THE GENERATOR is a policy change and must be re-scored, even though it never changes what the rulebook says.

The decisionWhen to reach for it