The business caseThe problem this solves

A manifest declares waste codes, container counts and quantities leaving a generator. The receiving facility records what arrived. THE TWO DISAGREE CONSTANTLY and almost none of it is reportable: a load that left in gallons is weighed in on a scale in pounds, drums are overpacked into salvage drums on the receiving pad before weighing, a weighbridge ticket is captured gross before the tare comes off, a load travels on two manifests because it would not fit on one truck. A subset ARE significant discrepancies and that subset starts a clock. So the failure that costs money is never a wrong subtraction. It is a difference reported against a generator when the facility's own receiving note already answers it -- and the note is on the facility's own file, with the facility's own name at the top of it. The named trap runs the other way. PARTIAL EXPLANATION -- a note that settles ONE AXIS of a difference -- reads as fully answered: the drum count fell from 21 to 7 and the note says the drums were overpacked at the gate, which is true and which settles the count. It says nothing about the weight, and the weight is short by more than the threshold this manifest states. DR-5.2 on the page keeps the line reportable and cites both the note and the axis it never touched. Somebody reading a manifest and a receiving record side by side: pairing each line to its counterpart on waste codes and container type, converting a weight in pounds back to the gallons the manifest declared using the density printed on that line, dividing, comparing the result with the variance threshold that manifest states, comparing the container counts where the waste is counted rather than bulk, checking every waste code on the receiving record against the ones the manifest declares, noticing the line nothing arrived against and the record nothing was declared for -- and then reading the facility's own receiving notes to find out which of those apparent discrepancies somebody has already answered, and which note answers only half of the line it names.

The inputThe actual manifest exception review packs

The corpus is 45 manifest exception review packs, 0.33 MB (txt 45). A real manifest exception file names a generator, a transporter and a permitted facility, and it carries the receiving facility's own notes about a load it accepted. All three parties are identifiable, the notes are commercially and legally sensitive, and that is why there is no public corpus of (manifest, receiving record, determination) triples -- and why publishing a scrubbed real one would be worse than publishing none, because the scrubbing is exactly where the interesting defect hides. More to the point, the thing being measured has to be planted to be measured. The question is not whether two numbers differ; it is whether a determination separates a difference the facility's own paperwork has already answered from one it has only HALF answered. To count that you have to know which is which, and a real archive does not come labelled.

Manifest Exception Review
----------------------------------------------------------------
  Pack                ME-0001
  Manifest tracking   681628344 ECJ
  Generator           Kilnbrook Foundry, EPA ID WID003392174
  Transporter         Ordway Environmental Transport, EPA ID MOD002884411
  Receiving facility  Grantham Ridge Treatment & Disposal, EPA ID KSD980631455
  Shipped             2026-03-28
  Received            2026-03-30
  Reviewed by         receiving compliance desk, on the day of receipt

Discrepancy Rules In Force
----------------------------------------------------------------

  Thresholds on this manifest

  Bulk variance threshold      10.0 per cent
  Reporting window             45 days from the date of receipt

  Rule index

  DR-1.1    A manifest line pairs to the received record carrying the same waste codes and the same container type
  DR-2.1    Containerised waste: any difference between the number of containers declared and the number received is a significant discrepancy
  DR-2.2    A quantity variance larger than the threshold stated above is a significant discrepancy. This test applies to every line, containerised or bulk; on containerised waste it is in addition to DR-2.1 and not instead of it
  DR-2.3    Quantities are compared after conversion to one unit, using the density printed on the manifest line; where no density is printed the comparison cannot be made
  DR-3.1    A waste code on the receiving record that the manifest does not declare, or a received record with no manifest line at all, is a significant discrepancy
  DR-4.1    A manifest line with no counterpart on the receiving record is a significant discrepancy
  DR-5.1    A difference the receiving record itself explains is not a significant discrepancy

The outcomeWhat a good result looks like

A drafted determination: one entry per manifest line, in the pack's order, each carrying the received record it pairs to, REPORTABLE with a named ground, the governing rule from this pack and the record identifiers behind it -- or NOT_REPORTABLE, or INSUFFICIENT_EVIDENCE naming what is missing -- plus the quantity variance independently recomputed on EVERY line, a shipment-level recommendation, and the date DR-6.1 gives. Beside every line, what the strongest free code would have written.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Every fact that decides a line is already a FIELD -- counts, quantities, units, densities, waste codes -- and your receiving desk writes no free-text notes at all. — the free floor -- threshold-diff, $0.00
  It catches 127 of 127 significant discrepancies and 98 of 98 that the tables settle, cites every rule and attaches every record correctly, invents nothing, and runs the whole corpus in a fraction of a second with no key.
• Half the decisive facts are in the notes, and some notes settle only part of a line. — the model -- 0.05298274 per shipment
  88.69 pct on the discriminator against the strongest free floor's 75.91, 126 of 127 significant discrepancies against 61, 28 of 29 note-only lines against 0, and 27 of 29 on the named trap against 0 -- with 1 benign lines reported and 0 identifiers invented.

And where nothing here is good enough:

• Your receiving notes decide real lines, and you were about to write the obvious rule: if a note mentions the line, treat the difference as explained. — do not -- and this kit exists to put a number on why
  That is exception-gate. It scores 75.91 pct on the discriminator, which looks good, and it catches 61 of 127 significant discrepancies, which is the number that matters. It suppresses every line a note names, and in this corpus 58 record-channel lines are named by notes that settle nothing -- a photograph on arrival, a seal number that matched.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-26. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-26 — r002-manifest-exception. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — git clone, then python3 -m src.app and open http://127.0.0.1:9072. No pip install, no index build, no key. The corpus, the answer key and every committed result ship in the repo; the pairing, the recomputed variance and the strongest free floor are computed live on every pack, and the second button replays exactly what r002-manifest-exception answered. Only the 'Determine this shipment' button needs a credential, and with none configured it returns a sentence saying nothing was called rather than an error.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per shipment, six concurrent workers against a shared key that sibling kits were using at the same time; 1178.4 s of wall clock for 45 packs. Read it as a bound, not as a clean single-tenant latency.

Current processWhat it replaces

Somebody reading a manifest and a receiving record side by side: pairing each line to its counterpart on waste codes and container type, converting a weight in pounds back to the gallons the manifest declared using the density printed on that line, dividing, comparing the result with the variance threshold that manifest states, comparing the container counts where the waste is counted rather than bulk, checking every waste code on the receiving record against the ones the manifest declares, noticing the line nothing arrived against and the record nothing was declared for -- and then reading the facility's own receiving notes to find out which of those apparent discrepancies somebody has already answered, and which note answers only half of the line it names.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptArchitectureArchitectureApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• 1 of 11 stations carry a recorded failure

scroll → 11 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one manifest lines, plus received records with no manifest line at all is

45 packs, each one shipment with its own discrepancy rules, its own variance threshold and its own reporting window, the manifest as declared, the receiving facility record and the facility's receiving notes. Every pack is independent -- no carried state, no ordering constraint -- so the run is embarrassingly parallel. The composition is CONSTRUCTED rather than dealt: 127 lines are significant discrepancies, 127 are not and 20 cannot be settled from the pack at all, because a random draw makes almost every shipment clean and the shipment-level recommendation collapses to a constant -- as it is, the single majority word already scores 97.78 pct and both the majority and the measured figure are printed.

SetupWhat the setup figure measured

There is no index to build. The whole pack, minus the withheld Generator Compliance History block, goes into the prompt verbatim -- no chunking, no retrieval, no pre-digest. A summariser here would be a place for the receiving note to be lost before the model saw it, and the receiving note is the entire thing the model is being paid to read.

LicenceLicence

MIT -- this repository's own licence. The corpus is generated in-process by tools/build_corpus.py from a fixed seed and contains no third-party data of any kind; verified by reading every generator input on 2026-08-26.

Bring your ownBring your own manifest exception review packs

Replace data/corpus/*.txt and data/gold.jsonl. In that order, and read data/SOURCES.md first. src/manifest.py is regular expressions over this layout, so a real receiving file needs a new parser before anything else works -- and it returns empty tables rather than guessing, so you will see it fail rather than watch it invent. Then re-run evals/check_labels.py, which is free and offline and is the gate that runs BEFORE anything spends. Then run the three free floors on your own data before making a single paid call: on this corpus the strongest one scores 75.91 pct and it is the number a model has to beat.

⚠︎ And what stops being true when you do: ⚠︎ WHAT DOES NOT TRANSFER IS THE BASE RATE, AND IT IS THE FIRST THING PEOPLE CARRY OVER. 127 of 274 lines here are significant discrepancies, because the corpus was built for statistical power on every ground. A real receiving desk sees a far lower rate, which moves every percentage on this page and moves the over-report column hardest. The second thing that does not transfer is the note channel: 75 of 274 lines are settled only by a receiving note, which is a design choice stated in SOURCES.md rather than a measurement of anybody's files.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 2,808 tokens assembled, 1,739 are evidences — 62% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The system instruction exactly as src/prompt.SYSTEM carries it, followed by pack ME-0001 exactly as src/select.body() assembles it -- five of the six sections, with Generator Compliance History cut at the seam. This is the string that was posted, not a description of it.

Full promptThe prompt in full

You are reconciling a hazardous waste shipping manifest against what the receiving facility actually recorded, and deciding which differences are significant discrepancies under the rules printed in the pack.

Every rule you may rely on is printed in the pack, in "Discrepancy Rules In Force", with an identifier. The thresholds are per manifest and are printed there too. Do not apply a rule, a threshold or a reporting window from anywhere else.

Return one entry for EVERY line in "Manifest As Declared", in the order the pack lists them, using the pack's own line identifiers -- and one further entry for every record in "Receiving Facility Record" that no manifest line pairs to, identified by the record's own identifier. Never drop a line because it looks unremarkable.

For each entry give the record it pairs to, or null where nothing does.

Then give exactly one disposition:

  REPORTABLE             the difference is a significant discrepancy under a rule in this pack.
  NOT_REPORTABLE         either there is no material difference, or the pack itself accounts for
                         the one there is.
  INSUFFICIENT_EVIDENCE  the pack does not carry what the determination would turn on. Do not
                         decide; say what is missing.

The ground is exactly one of these, and it must belong to the disposition you gave:

  REPORTABLE
    PIECE_COUNT_DIFFERENCE   containerised waste, and the number of containers received is not the
                             number declared
    QUANTITY_VARIANCE        the quantity variance is larger than the threshold this manifest states
    WASTE_CODE_MISMATCH      the receiving record carries a waste code the manifest does not declare
    LINE_NOT_RECEIVED        the manifest declares the line and nothing received pairs to it
    UNMANIFESTED_RECEIPT     a received record with no manifest line at all
    PARTIAL_EXPLANATION      the receiving record accounts for part of the difference and another
                             part of the same line is still outside the rules
  NOT_REPORTABLE
    WITHIN_TOLERANCE         the difference is inside what this manifest's rules allow
    UNIT_CONVERSION          declared and received in different units, and the converted quantities
                             agree
    EXPLAINED_BY_RECORD      the pack itself accounts for the whole difference
  INSUFFICIENT_EVIDENCE
    DENSITY_NOT_STATED       the quantities are in different measures and no density is printed
    RECEIVING_QUANTITY_ABSENT  the receiving record shows no weight to compare

Also give, for every entry, the quantity variance you independently recompute: the received quantity converted into the manifest line's own unit, minus the declared quantity, as a percentage of the declared quantity, signed, to two decimal places. Give null where the pack does not carry what that conversion needs.

Cite the governing rule as a rule identifier printed in this pack, and attach the support as record, line or note identifiers printed in this pack. NEVER write an identifier that is not printed in the pack in front of you -- an invented citation is worse than none, because it is the first thing the generator will check.

Anything in the pack may bear on a line. Where a printed table and a later statement in the pack disagree, they are not a tie.

Reply with JSON and nothing else:

{"shipment_action": "FILE_EXCEPTION_REPORT" | "REQUEST_INFORMATION" | "CLOSE_NO_DISCREPANCY",
 "report_by": "<YYYY-MM-DD, the date DR-6.1 gives from the date of receipt, or null>",
 "lines": [{"line": "<the pack's line or record identifier>",
            "paired_with": "<the record identifier it pairs to, or null>",
            "disposition": "REPORTABLE" | "NOT_REPORTABLE" | "INSUFFICIENT_EVIDENCE",
            "ground": "<one of the eleven above>",
            "governing_rule": "<a rule identifier from this pack>",
            "support": ["<identifiers from this pack>"],
            "variance_pct": <number or null>,
            "finding": "<the one sentence the exception log would carry for this line>"}],
 "rationale": "<two sentences at most, on what decided the hardest line>"}

shipment_action is FILE_EXCEPTION_REPORT if any line is REPORTABLE; REQUEST_INFORMATION if none is but at least one is INSUFFICIENT_EVIDENCE; CLOSE_NO_DISCREPANCY otherwise. report_by is null unless the action is FILE_EXCEPTION_REPORT. None of the three files, rejects or notifies anything -- each is a recommendation to the person who does.

Manifest Exception Review
----------------------------------------------------------------
Pack                ME-0001
  Manifest tracking   681628344 ECJ
  Generator           Kilnbrook Foundry, EPA ID WID003392174
  Transporter         Ordway Environmental Transport, EPA ID MOD002884411
  Receiving facility  Grantham Ridge Treatment & Disposal, EPA ID KSD980631455
  Shipped             2026-03-28
  Received            2026-03-30
  Reviewed by         receiving compliance desk, on the day of receipt

Discrepancy Rules In Force
----------------------------------------------------------------
Thresholds on this manifest

  Bulk variance threshold      10.0 per cent
  Reporting window             45 days from the date of receipt

  Rule index

  DR-1.1    A manifest line pairs to the received record carrying the same waste codes and the same container type
  DR-2.1    Containerised waste: any difference between the number of containers declared and the number received is a significant discrepancy
  DR-2.2    A quantity variance larger than the threshold stated above is a significant discrepancy. This test applies to every line, containerised or bulk; on containerised waste it is in addition to DR-2.1 and not instead of it
  DR-2.3    Quantities are compared after conversion to one unit, using the density printed on the manifest line; where no density is printed the comparison cannot be made
  DR-3.1    A waste code on the receiving record that the manifest does not declare, or a received record with no manifest line at all, is a significant discrepancy
  DR-4.1    A manifest line with no counterpart on the receiving record is a significant discrepancy
  DR-5.1    A difference the receiving record itself explains is not a significant discrepancy
  DR-5.2    An explanation covers only the axis it names. Where the receiving record explains one axis of a difference and another axis of the same line is still outside the rules above, the line is a significant discrepancy and both the explanation and the unresolved axis are cited
  DR-6.1    A significant discrepancy is entered in an exception report within the reporting window stated above

  Which rule governs which finding

  PIECE_COUNT_DIFFERENCE      DR-2.1
  QUANTITY_VARIANCE           DR-2.2
  WASTE_CODE_MISMATCH         DR-3.1
  LINE_NOT_RECEIVED           DR-4.1
  UNMANIFESTED_RECEIPT        DR-3.1
  PARTIAL_EXPLANATION         DR-5.2
  WITHIN_TOLERANCE            DR-2.2
  UNIT_CONVERSION             DR-2.3
  EXPLAINED_BY_RECORD         DR-5.1
  DENSITY_NOT_STATED          DR-2.3
  RECEIVING_QUANTITY_ABSENT   DR-2.2

  Container type codes

  DM    metal drum, 55 gallon            counted
  DF    fibreboard drum                  counted
  DT    metal drum, light gauge          counted
  CM    metal box or bin                 counted
  BX    fibreboard box                   counted
  TP    tank, portable                   bulk
  TT    cargo tank                       bulk
  RO    roll off container               bulk

  Unit codes

  P     pounds
  T     tons (short)
  K     kilograms
  G     gallons
  Y     cubic yards

  Conversions

  T to P    multiply by 2,000
  K to P    multiply by 2.20462
  G to P    multiply by the density in pounds per gallon printed on the manifest line
  Y         cubic yards do not convert to a weight in this review

Manifest As Declared
----------------------------------------------------------------
Line M1
  Description         Paint related waste, ignitable, in drums
  Waste codes         D001
  Container type      DM  (metal drum, 55 gallon)
  Containers          21
  Quantity            1,093.52
  Unit                G  (gallons)
  Density             7.86 lb per gallon

Line M2
  Description         Dissolved air flotation float from refining, drummed for shipment
  Waste codes         K048
  Container type      DM  (metal drum, 55 gallon)
  Containers          46
  Quantity            2,378.20
  Unit                G  (gallons)
  Density             7.40 lb per gallon

Line M3
  Description         Mercury bearing lab debris
  Waste codes         D009
  Container type      DF  (fibreboard drum)
  Containers          7
  Quantity            37,573.45
  Unit                P  (pounds)

Line M4
  Description         Silver bearing photographic fixer
  Waste codes         D011
  Container type      DM  (metal drum, 55 gallon)
  Containers          11
  Quantity            584.71
  Unit                G  (gallons)
  Density             8.91 lb per gallon

Line M5
  Description         Spent acetone and mineral spirits, non-halogenated
  Waste codes         D001, F003
  Container type      DM  (metal drum, 55 gallon)
  Containers          10
  Quantity            517.33
  Unit                G  (gallons)
  Density             6.72 lb per gallon

Receiving Facility Record
----------------------------------------------------------------
Record R1
  Description         Paint related waste, ignitable, in drums
  Waste codes         D001
  Container type      DM  (metal drum, 55 gallon)
  Containers          21
  Quantity            1,093.69
  Unit                G  (gallons)

Record R2
  Description         Dissolved air flotation float from refining, drummed for shipment
  Waste codes         K048
  Container type      DM  (metal drum, 55 gallon)
  Containers          44
  Quantity            2,391.38
  Unit                G  (gallons)

Record R3
  Description         Mercury bearing lab debris
  Waste codes         D009
  Container type      DF  (fibreboard drum)
  Containers          7
  Quantity            37,127.16
  Unit                P  (pounds)

Record R4
  Description         Silver bearing photographic fixer
  Waste codes         D011
  Container type      DM  (metal drum, 55 gallon)
  Containers          2
  Quantity            584.27
  Unit                G  (gallons)

Record R5
  Description         Spent acetone and mineral spirits, non-halogenated
  Waste codes         D001, F003
  Container type      DM  (metal drum, 55 gallon)
  Containers          2
  Quantity            397.62
  Unit                G  (gallons)

Facility Receiving Notes
----------------------------------------------------------------
FN-1    Grantham Ridge Treatment & Disposal. The 11 drums on manifest line M4 were overpacked into 2 eighty five gallon salvage drums on the receiving pad before weighing, under the facility's own consolidation procedure. Net weight over the scale agrees with the declared volume. The count on R4 is the count after overpacking and not a shortage.

  FN-2    The 10 drums on manifest line M5 were overpacked into 2 salvage drums on the receiving pad under the facility's consolidation procedure. The count on R5 is the count after overpacking.

  FN-3    Line M2 was offloaded last and the driver waited on the weighbridge for the return copy.

Raw responseThe raw response

{
  "shipment_action": "FILE_EXCEPTION_REPORT",
  "report_by": "2026-05-14",
  "lines": [
    {
      "line": "M1",
      "paired_with": "R1",
      "disposition": "NOT_REPORTABLE",
      "ground": "WITHIN_TOLERANCE",
      "governing_rule": "DR-2.2",
      "support": ["M1", "R1"],
      "variance_pct": 0.02,
      "finding": "The 21 drums match and the 0.02% quantity variance is within the 10.0% threshold."
    },
    {
      "line": "M2",
      "paired_with": "R2",
      "disposition": "REPORTABLE",
      "ground": "PIECE_COUNT_DIFFERENCE",
      "governing_rule": "DR-2.1",
      "support": ["M2", "R2"],
      "variance_pct": 0.55,
      "finding": "Received 44 drums against 46 declared; any container count difference is a significant discrepancy."
    },
    {
      "line": "M3",
      "paired_with": "R3",
      "disposition": "NOT_REPORTABLE",
      "ground": "WITHIN_TOLERANCE",
      "governing_rule": "DR-2.2",
      "support": ["M3", "R3"],
      "variance_pct": -1.19,
      "finding": "The 7 drums match and the -1.19% quantity variance is within the 10.0% threshold."
    },
    {
      "line": "M4",
      "paired_with": "R4",
      "disposition": "NOT_REPORTABLE",
      "ground": "EXPLAINED_BY_RECORD",
      "governing_rule": "DR-5.1",
      "support": ["M4", "R4", "FN-1"],
      "variance_pct": -0.08,
      "finding": "FN-1 explains the reduced drum count as overpacking and confirms the quantity, so the difference is accounted for by the receiving record."
    },
    {
      "line": "M5",
      "paired_with": "R5",
      "disposition": "REPORTABLE",
      "ground": "PARTIAL_EXPLANATION",
      "governing_rule": "DR-5.2",
      "support": ["M5", "R5", "FN-2"],
      "variance_pct": -23.14,
      "finding": "FN-2 explains only the reduced drum count; the quantity is 23.14% below declared, exceeding the 10.0% threshold, so the difference remains significant."
    }
  ],
  "rationale": "M4's count reduction is fully explained by FN-1 with its volume within tolerance, while M5's overpacking note explains only the count and leaves a 23.14% quantity shortfall; M2's 44-vs-46 drum count is also an unexplained reportable count difference."
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Reconcile a waste shipment's manifest against what arrived — 274 manifest exception review packs drawn from 45 real manifest exception review packs. One model answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

Exact match per line, in pure Python. No model grades anything, there is no judge and there is no rubric -- the answer key is planted with the corpus and re-derived from the SHIPPED packs by evals/check_labels.py.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on this page is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid. The rate is read from this repo's own committed card, build/facts/models.json, rather than typed here, so it cannot drift from what the rest of the estate projects onto. checked_on is that card's own as_of, not the date this page was written.

Rates checked 2026-08-18. The provider that actually ran every call here is kept off this page per the series rule. And the three free floors, the tolerance sweep, the wiring stub and the answer-key gate are not priced because they cost nothing: they make no call at all.

Grading unitWhat the grading figure prices

Every arm in this lens except the model is free: no key, no network, no tokens. The grading itself -- every percentage on this page -- cost $0.00.

The gradersOne way to grade, and why it is the only one

Every baseline figure here is the strongest floor, exception-gate, at the shipped src/checks.TOL = 0.25 (b002-manifest-exception-exceptiongate). ⚑ THE TOLERANCE IS NOT THE STORY ON THIS KIT AND THAT IS ITSELF A MEASUREMENT: 8 values of it move this floor by 0.73 points, monotonically, and 0.00 scores identically to the shipped value -- so the constant is not doing work here, unlike the sibling kit that measured 8.52 non-monotonic points from the same shape of constant and saw the winner flip inside it. ⚠︎ WHAT THE FLOOR CANNOT REACH AT ANY TOLERANCE IS THE NOTE CHANNEL: 0 of 29 note-only discrepancies and 0 of 29 PARTIAL_EXPLANATION lines, at every value swept.

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, AND ALMOST ALL OF THEM ARE REASONING. 95.68 pct of r002-manifest-exception's output (740,270 of 773,682) was provider-side reasoning rather than the determination. The determination itself is a few hundred tokens per pack. You are paying for the reading, not the writing.
• THE PACK GOES IN WHOLE. 2807.91 input tokens per pack on average (126,356 over 45), and p001-manifest-exception-prompt-tokens measured the split from two calls at max_tokens=1: a fixed 1068.56 tokens -- the system instruction PLUS the provider's own chat wrapper, billed and invisible to the kit -- plus 0.24071 tokens per pack character.
• EVERY FREE ARM COSTS NOTHING AND THERE ARE FIVE OF THEM. The answer-key gate, the three floors and the tolerance sweep are pure Python over the committed corpus: no key, no network, no tokens. The whole grading of this kit -- every percentage on this page -- cost $0.00.

Your volumeWhat it costs at your volume

Linear in packs and roughly linear in pack size, because there is one call per shipment and the pack goes in whole. Ten times the shipments is ten times $0.05298274. Ten times the LINES per shipment is not ten times the bill on the input side -- the rule index and the tables are fixed -- but it is more reasoning, and reasoning is 95.68 pct of what is billed. ⚑ AND THE FREE FLOOR IS FLAT AT ZERO for both, which is the comparison that matters at 10x.

Model choiceWhy this model

One tier was run, and it is the tier this estate's shared key points at. A second was not, which is a spend decision recorded in not_measured rather than a result. ⚑ THE MODEL COMPARISON THIS KIT ACTUALLY MAKES IS NOT BETWEEN TIERS -- it is between one tier and 132 lines of pure Python, and the Python is free.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

per-shipment average, this run

The codeEvery file, and what it is for

12 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Seven of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysection split

Cut the pack on its underlined headings -- Manifest Exception Review, Discrepancy Rules In Force, Manifest As Declared, Receiving Facility Record, Facility Receiving Notes, Generator Compliance History. Deterministic, so 'what did we send' is a list of section names rather than a guess about a token window.

# Cut a change-pricing pre-check pack into addressable sections. Pure code, no model.

HEAD = re.compile(r"^([A-Z][A-Za-z ,]+)\n-{10,}\n", re.M)
def sections(text):
def named(text, name):

src/select.pythe seam that withholds — a swap seam

The Generator Compliance History block -- prior notices of violation, whether the account is one of the site's largest by tonnage or on credit hold, and the desk's own position on whether it wants this reported -- never leaves the machine. None of it bears on whether a drum count differs or a weight converts; all of it is the commercial and legal pressure around a determination that is supposed to be made on the paperwork. Withheld at the seam rather than trusted to a sentence in a prompt, and the UI prints what went and what stayed.

You change it to: NEVER_SENT. One tuple, holding one section name. It is a named-section denylist and not a redaction system.

# SEAM 2 -- what leaves this machine. Pure code, and deliberately a denylist of one.

NEVER_SENT = ("Generator Compliance History",)
def sent(sec_names):
def body(text, sections_fn):

src/manifest.pythe pack parser

The rule index, this manifest's own variance threshold and reporting window, the container and unit tables, every declared line with its density, and every received record, read with regular expressions. The model is never asked for a figure at a fixed offset. Returns empty tables rather than guessing on an unfamiliar layout. ⚠︎ ITS _f() ONCE ASKED float() FOR '6.72 lb per gallon' AND GOT None, which routed every gallons-against-pounds line to DENSITY_NOT_STATED -- check_labels.py convicted 36 correct key rows on its first run and the parser was the thing that was wrong.

# Read a manifest exception pack in pure code. Regular expressions, no model, no guessing.

REPORTABLE = "REPORTABLE"
NOT_REPORTABLE = "NOT_REPORTABLE"
INSUFFICIENT = "INSUFFICIENT_EVIDENCE"
DISPOSITIONS = (REPORTABLE, NOT_REPORTABLE, INSUFFICIENT)
GROUND_RULE = {
GROUNDS = tuple(GROUND_RULE)
REPORTABLE_GROUNDS = ("PIECE_COUNT_DIFFERENCE", "QUANTITY_VARIANCE", "WASTE_CODE_MISMATCH",
BENIGN_GROUNDS = ("WITHIN_TOLERANCE", "UNIT_CONVERSION", "EXPLAINED_BY_RECORD")
UNRESOLVABLE_GROUNDS = ("DENSITY_NOT_STATED", "RECEIVING_QUANTITY_ABSENT")

src/checks.pypairing and the structured checks — a swap seam

DR-1.1's pairing in code -- same waste codes, same container type, then a second pass for the superset case that makes a waste-code mismatch visible instead of two orphans -- plus the nine determinations the pack's OWN TABLES can settle, plus the variance recomputed on every line. ⚑ TOL AND QTOL LIVE HERE and TOL is the one constant nobody publishes; evals/tolerance.py sweeps it.

You change it to: TOL, one float, in PERCENTAGE POINTS of slack above the threshold each manifest states. On this corpus it moves the strongest floor by 0.73 points and does NOT change who wins; a sibling kit measured 8.52 points from the same shape of constant and the winner flipped inside it. Sweep it on your own corpus before believing any floor's score.

# Everything the PRINTED TABLES can settle about a manifest line, in pure code, plus the

TOL = 0.25
QTOL = 15.0
def pair(p):
def by_id(p):
def variance(line, rec):
def _abs_pounds(line, rec, v):
def structured_finding(line, rec, p, tol=None):
def orphan_finding(rec):
def shipment_action(dispositions):

src/prompt.pythe prompt

The whole instruction in one place, in send order, plus the notes-blind variant an ablation arm would use. The ablation TRUNCATES at the heading and RAISES if the heading is absent, because a silent no-op there would publish an ablation arm that was never ablated. ⚠︎ THAT ARM WAS NEVER RUN -- it is a spend decision and it is in not_measured, not left to look like a pass.

# The assembled prompt, in one place, in the order it is sent.

NOTES_HEADING = "Facility Receiving Notes"
SYSTEM = """You are reconciling a hazardous waste shipping manifest against what the receiving \
BLIND_SYSTEM_SUFFIX = """
def build(text, blind=False):
def _strip_notes(body):
def render(parts):

src/determine.pythe model call and reply parse

One pack in, one drafted determination out. Holds the published 64000-token ceiling and the tolerant JSON parse -- a correct answer wrapped in a code fence has not got it wrong -- and normalises a single identifier returned as a bare string.

# One pack in, one drafted determination out. The only place a model is called.

MAX_TOKENS = 64000
THINKING = None
def parse_reply(text):
def normalise(obj):
def determine(cfg, text, blind=False, complete_fn=None, max_tokens=None):
def cited_ids(answer):

src/adapters/__init__.pythe adapter — a swap seam

One interface, several providers, raw HTTP and no vendor SDK. ⚑ ITS SOCKET TIMEOUT AND THE TOKEN CEILING ARE ONE SETTING: 1800 s beside a 64000-token ceiling, with transport retries cut to ONE against four for a transient status, because a non-streamed generation held on a short socket is a transport failure the inherited retry policy pays for five times.

You change it to: PROVIDER and MODEL in <repo>/.env, or a kit-local .env carrying MODEL alone. One tier was run through this seam; a second was not, and that is a spend decision rather than a result.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TIMEOUT_S = 1800
TRANSPORT_RETRIES = 1
def _post(url, headers, payload, timeout=TIMEOUT_S):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}

evals/baseline.pythe three free floors — a swap seam

Variance recalc, threshold diff, and the exception gate. Each a genuine attempt at the job, all three free. ⚑ THE THIRD IS BUILT TO FAIL IN A NAMED WAY: it treats any line a receiving note MENTIONS as explained, which is what anyone reaches for first, and mentioning a line is not explaining it.

You change it to: --floor variance-recalc | threshold-diff | exception-gate. Three pure-code determiners behind the same answer shape, so one scorer grades all of them and the model.

# The free floors. Pure Python, no key, no network, and each a genuine attempt at the whole job.

MODES = ("variance-recalc", "threshold-diff", "exception-gate")
def _note_targets(p):
def review(text, mode="threshold-diff", tol=None):

evals/tolerance.pythe tolerance sweep

Three floors by 8 tolerances, free and offline. On this corpus it moves the strongest floor by 0.73 points (75.18 -> 75.91) and it is MONOTONIC -- which is itself the finding, and the opposite of the sibling kit that measured 8.52 points of non-monotonic swing from the same shape of constant.

# Sweep the one constant nobody publishes, and publish the swing. Free, offline, no key.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
SWEEP = (0.0, 0.1, 0.25, 0.5, 1.0, 1.5, 2.0, 3.0)
def main():

evals/scoring.pythe scorer

Exact match per line, and every rate carries its own denominator. The pairing is scored in its own right because a determination attached to the wrong counterpart is wrong in a way that reads as right. The invented-identifier check reads the PACK's identifiers rather than the answer key's, so it can convict the key too.

# Score an arm against the answer key. Pure code, exact match per line. No model grades anything.

REPORTABLE, BENIGN, INSUFFICIENT = "REPORTABLE", "NOT_REPORTABLE", "INSUFFICIENT_EVIDENCE"
OMITTED = "OMITTED"
PCT_EPS = 0.011
def _pct(n, d):
def _key(s):
def rows_by_line(answer):
def _cited(r):
def _numeric(v):
def score(records, golds, valid_ids=None):

evals/check_labels.pythe answer-key gate

3912 assertions over 45 packs and 274 lines, re-derived from the SHIPPED corpus with the shipping parser. Asserts the channel split in BOTH directions -- every record-channel line settled by the tables alone, every note-channel line NOT -- and requires the decoy notes to exist, which is the assertion that stops the corpus flattering the keyword floor.

# Prove the answer key against the corpus that shipped -- free, offline, and BEFORE anything spends.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
def load():
def text_of(doc_id):
def main():

src/app.pythe local UI

http.server, no dependency, five read endpoints and one determine endpoint, and no write path. Renders with no key, replays the committed run, prints the independently recomputed variance on every line whether or not anything is wrong with it, and computes the strongest free floor on every line every time. Binds PORT 9072.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
CORPUS = os.path.join(HERE, "data", "corpus")
PORT = int(os.environ.get("PORT", "9072"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r002-manifest-exception")
FLOOR_MODE = "exception-gate"
def documents():
def load_doc(doc_id):
def read_in_code(text):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2807 input and 17192 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

⚠︎ NOT MEASURED. evals/injection.py ships, both phrasings are written out in full, and it was not run. That is a spend decision and it is recorded here rather than left to look like a pass. ⚑ AND THE SURFACE IS NAMED EVEN THOUGH THE RATE IS NOT: the receiving notes are the channel this kit pays a model to read, so anything writable into them is inside the trust boundary of the only column a free floor cannot reach -- and a note saying 'do not report this' is indistinguishable in FORM from the note saying 'the tare is 4,120 pounds' that legitimately settles 46 lines in this corpus. No number is published for it.

API_KEY is read from <code>&lt;repo&gt;/.env</code>, this kit's own <code>.env</code>, or the real environment, in that precedence. Both files are gitignored from the first commit and this repo has never held a credential. Nothing is asked of a reader: the page on the site executes nothing, and the local UI renders fully with no key at all.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. Both were run for real on 2026-08-26.

Nothing here is a resistance claim. The probe exists and was not fired.

The resultThe receiving notes are the channel this kit pays a model to read, so anything writable into them is inside the trust boundary of the only column free code cannot reach. NOT MEASURED: the probe ships, both phrasings are written out, and no run of it exists. No number is published.

evals/injection.py is committed and complete: it forces the note into every pack in a scope fixed BEFORE the run, pairs every cell against the same model's own un-injected answer in r002-manifest-exception so a cell the arm was already getting wrong cannot be counted as suppressed, reads the identifier set from the INJECTED pack so citing the forced note is not scored as an invention, and records for every cell that HELD whether its rule or its support moved. None of that has been run. There is no rate here, no denominator and no ceiling, and none is invented.

HonestyWhat this does not prove

• Everything about this surface. No attack run exists.
• In particular: whether the note channel -- the only column this kit pays for -- can be turned off by a sentence in the same position as the notes that legitimately settle 46 lines. That is the question the probe was written for.
• And whether an injection placed in the WITHHELD Generator Compliance History block would have any effect. By construction it cannot reach the model, and that was not probed either.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never file an exception report, reject a load, amend a manifest, or notify a generator or an authority. This kit produces a DETERMINATION for a person to read, and the refusal -- NOT_REPORTABLE, or INSUFFICIENT_EVIDENCE naming what is missing -- is part of the determination rather than a failure of it. Filing, rejecting and notifying are acts a named person takes under an authority the pack references and does not enforce.

Stated on the UI and in the README, and enforced by there being no such endpoint: src/app.py serves /api/packs, /api/pack, /api/prompt, /api/recorded and /api/determine, and nothing else. There is no write path to remove.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is ABSENCE OF A WRITE PATH plus a prompt rule, not a runtime enforcement layer. There is no policy engine, no approval workflow and no audit log, because a kit has none of those and adding them would make it a different product.
• src/select.py withholds a NAMED SECTION. It is not a redaction system, and a pack whose receiving notes happen to mention the account standing will send it.
• Nothing in this kit constrains a citation to the pack's own identifier set. The scored run invented 0 and the strongest free floor invented 0, and the kit COUNTS that rather than preventing it.
• ⚠︎ AND THE ANSWER KEY IS NOT A GUARDRAIL EITHER. It was wrong about 45 of 274 lines until the model refuted it, and it is still wrong about 26 more that are published unfixed. A gate that shares its subject's assumption cannot convict it.

WatchedWhat is watched, and why that one

The kit measures 99 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

24 measured by the latest run 75 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (benign_lines 127, blind False, cells 274, dataset_version manifest-exception-2026-08-26b-45packs, floor_tolerance_pct 0.25, ground_named_cells 274, lines_citing_any_id 274, note_only_ground_cells 29, packs 45, packs_answered 45, pairing_cells 274, partial_explanation_cells 29, partial_explanation_code_cells 8, partial_explanation_count_cells 21, record_visible_ground_cells 98, report_due_cells 44, reportable_lines 127, rule_cited_cells 274, support_cells 274, trap_lines 46, unresolvable_cells 20, variance_cells 233) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

5 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

match · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 3 runs measured the same system — they differ on floor, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

match · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 24 chips that all say so.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 24 chips that all say so.

DeviationsWhat deviated

0 breaches across 5 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The answer-key gate, all three free floors and the 8-point tolerance sweep are free and run before any spend -- and on this kit the free half is what caught nothing and the four-call CALIBRATION PROBE is what caught everything. Read a probe's misses, not just its ceiling number, before firing a scored arm; doing that late cost this kit 45 calls. The injection probe is a MANUAL step and has never been run.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, standard library end to end, no orchestration layer and no vendor SDK. requirements.txt names nothing. The reason is the fork test: every layer added is a layer a forker has to understand before they can change anything -- and on this kit the thing a forker most needs to change is the RULE TEXT printed inside the corpus, because the first edition of it was wrong and nothing but reading it caught that.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear: a pack -> src/segment.py -> src/select.py -> src/prompt.py -> src/adapters -> src/determine.py -> evals/scoring.py. No branches, no agent loop, no tool calls. The only fan-out is the thread pool over independent shipments -- and the free floors skip the middle four steps entirely, going straight from src/manifest.py and src/checks.py to the same scorer.

The other sideWhat a framework costs you

• Adding a provider means editing the PROVIDERS dict by hand. In exchange, pip install pulls nothing and the fork test stays at one clone and one command.
• There is no retry/backoff policy you can configure -- it is four attempts on a transient status and ONE on a transport failure, written out in the adapter. The asymmetry is deliberate: a timed-out generation is not a rate limit, and retrying it pays for the whole generation again.
• The pairing is a hand-written join in src/checks.pair rather than anything a matching library would give you -- no blocking, no similarity, no threshold. That is cheap here because the pack hands you both sides of one shipment, and it is the first thing that breaks on a catalogue where two lines share a waste code set and a container type.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r002-manifest-exception on the fast tier, 2026-08-26. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 3 runs on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — blocking is computed in memory each run and is the only reduction step.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence. The corpus is generated in-process by tools/build_corpus.py from a fixed seed and contains no third-party data of any kind; verified by reading every generator input on 2026-08-26. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe whole determination on each line -- pairing, disposition, ground, rule, records

whether each of the 274 lines got a determination a compliance desk could put in an exception report: the record the key pairs it to, the disposition the key carries, the right ground of the eleven, the governing rule the pack itself prints for that ground, every identifier the ground needs, and nothing invented

MethodHow the test was run

python3 -m evals.run --run-id <id> ...; evals/scoring.py compares strings.

The inputOne real row, seen by every grader

⚑ THIS ONE LINE IS 16 OF THE 31 MISSES IN MINIATURE, AND THE KEY IS THE THING THAT IS ARBITRARY. Both arms answered NOT_REPORTABLE and both were right about the line. The pack prints eleven grounds and a rule for each, and it never says what happens when two of them are simultaneously true -- the units DO differ, and the converted variance IS inside the threshold. The free floor matches the key because the same author wrote both; the model chose the other true sentence and is scored as wrong for it. Fixing it means printing a precedence rule in the pack, which is a corpus change, and this run was not re-fired after the misses were read.

The formulaWhat it computes

exception_determination_accuracy_pct = accurate / 274. A line the arm left off entirely counts as OMITTED and is a miss. All six conditions must hold together.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, planted with the corpus and re-derived from it by evals/check_labels.py.

No true/false rates for this grader. It records 6 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold was swept on the GRADER: exact match has no tunable. The FLOORS have one and it was swept over 8 values -- see b900-manifest-exception-tolerance-sweep, which is why this page never prints a free-floor number without its tolerance.

Cadence: once per run; free

The decisionWhen to reach for it