The business caseThe problem this solves

A fuel surcharge is the second-largest line on most freight bills and it is the one nobody checks, because checking it means re-deriving a number rather than reading one. The surcharge is a percentage from a bracket table; the bracket is chosen by a published index price; and the index price is chosen by a LAG RULE applied to the shipment date -- the most recent Sunday on or before it, then back a further N whole weeks, where N is on the contract and is not the same on every contract. Apply the wrong week and the result is a REAL percentage from a REAL published row applied to the RIGHT base. It sits inside any tolerance band, its arithmetic agrees with the basis printed on the invoice, and nothing downstream of it can see the error. Today this is caught by somebody re-deriving one line by hand, or not at all. Replaces re-deriving a surcharge line by hand against a published series, and replaces the carrier-basis check most freight-payment systems already run -- which verifies that the arithmetic on the line agrees with the index the carrier printed on it, and therefore cannot see a wrong week at all.

The inputThe actual carrier fuel invoices

The corpus is 50 carrier fuel invoices, 0.12 MB (txt 50). A real carrier fuel invoice is not something this repo can publish: it names a real carrier, a real shipper's lane structure and negotiated linehaul rates line by line, and a named billing contact with a direct extension. There is no public corpus of (freight invoice, correct recalculation) pairs. Generating it also makes the label mechanical. Gold's status, applicable week, published price, percentage, recomputation and variance are ALL src/schedule.py run over the same values the invoice states under that invoice's own contract terms -- never somebody's reading of the carrier's remark, and never the basis the carrier claims -- and evals/check_labels.py re-runs the rule over every one of the 332 lines, and drives the lag rule directly on each of them, before any run may spend. The composition is an exact count over fifteen buckets rather than a sampled proportion, and each bucket is a reading a careless recompute gets wrong. Four of them are the point: 54 lines rated off a real published week that is not the one the contract selects (the case an amount-based check cannot see), 20 lines whose selected week was NEVER PUBLISHED, 22 capped lines and 20 all-in lanes (the two differences that are supposed to be there, and the false alarms that get a worklist ignored).

Invoice
-------
INV-2421-NJ

Invoice Date
------------
2026-04-17

Fuel Surcharge Schedule
-----------------------
FSC-1 -- illustrative, shipped with this kit; reproduces no real carrier tariff and no real published index

Contract Terms
--------------
fuel surcharge lag        1 week(s) -- whole weeks, stepped back from the shipment's own index week
maximum surcharge         32.00 pct
surcharge base            linehaul charge only; accessorial charges are not surcharged
index series              IDX-1, published weekly, each row covering the week ending on the Sunday named

Shipment Lines
--------------
amounts are in the trading currency's major unit; the index is stated to three decimals
SHP-30132   Port Gate 1 to Depot 3         shipped 2026-03-23      linehaul 4209.25         accessorials 324.25   basis tariff
SHP-30416   DC 1 to Plant 1                shipped 2026-02-24      linehaul 1390.25         accessorials 176.75   basis tariff
SHP-30175   DC 2 to Plant 2                shipped 2026-03-31      linehaul 4072.75         accessorials 350.00   basis all-in
SHP-30606   Plant 2 to DC 2                shipped 2026-03-23      linehaul 4128.75         accessorials 0.00     basis tariff
SHP-30787   DC 6 to Depot 3                shipped 2026-03-15      linehaul 4765.00         accessorials 535.00   basis tariff
SHP-30037   DC 6 to Depot 5                shipped 2026-03-27      linehaul 3407.25         accessorials 310.75   basis tariff
SHP-30761   Depot 2 to Yard 1              shipped 2026-02-13      linehaul 3865.00         accessorials 0.00     basis tariff

Fuel Surcharge As Billed
------------------------
SHP-30761   carrier index 4.055 at 34.00 pct       billed fuel surcharge 1314.10
SHP-80039   carrier index -- not applicable        billed fuel surcharge 0.00

The outcomeWhat a good result looks like

One recovery worklist per invoice: which lines were rated off a different index row than the contract selects, WHICH ROW that was and which one should have been used, what the difference is worth in minor units, whether the pattern is one wrong basis applied across the invoice or a scattering of one-offs, and what the invoice's total claim would come to. A person raises it.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• You already run a freight-bill audit that checks the carrier's own stated basis — this kit, and read the wrong-week recall only
  Your existing check is the floor on this page: 77.71 pct of statuses right, zero false alarms, and 0.0 pct on the population that actually costs money. The whole value of adding a recomputation is the 54 wrong-week lines and the 20 lines whose week was never published, and those two numbers are the only ones worth carrying across.
• You want the lag rule and the bracket lookup done reliably, at volume, cheaply — not this shape -- derive the week and join the row in CODE
  The lag rule is four lines of date arithmetic and the bracket is a table lookup; src/schedule.py already does both, deterministically and for free. This kit asks the MODEL to do them so that the derivation can be MEASURED, and it measured 100.0 pct on the row. 2162 of every call's 6188 input tokens are the table, and 93.3 pct of the output is a reasoning pass doing arithmetic. A production build reads the invoice with the model and derives everything else in code.
• You want a number you can take to a carrier — this kit's row citation, and nothing else on the page
  A recovery conversation is not 'your surcharge looks wrong'. It is 'the contract's 1- or 2-week lag selects the week ending X, published at Y, which is Z pct on this linehaul; you billed off the week ending W'. That sentence is what src/schedule.py emits as computed_reason, and every element of it is on the page beside the verdict.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-23. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-23 — r001-fuel-surcharge. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with NO key configured reproduces everything on this page except a new call: the 50 invoices, the 50 gold rows, all five committed run records, and the local UI, which renders and returns a plain sentence saying nothing was called. python3 tools/build_corpus.py rebuilds the corpus byte-identically from the seed; python -m evals.check_labels and python -m evals.run --baseline both run for free and reproduce the floor's numbers exactly. What a clone cannot do without a key is recompute a NEW invoice or re-score the model.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per invoice, measured in evals/run.py around the whole extract call -- segment, select, prompt, the one model call, parse, span, and the pure-code lag rule, bracket lookup and arithmetic. There is no retrieval step to exclude, so this figure and the model-only figure in Cost are the same measurement; both are stated so a reader does not have to assume it. Measured at 12 concurrent workers, so it is per-call latency and not throughput: the whole 50-invoice run finished in 723.2 seconds. The slowest single call took 474446 ms, 5.5x the median -- on a task that is 93.3 pct provider-side reasoning, the tail is long.

Current processWhat it replaces

Replaces re-deriving a surcharge line by hand against a published series, and replaces the carrier-basis check most freight-payment systems already run -- which verifies that the arithmetic on the line agrees with the index the carrier printed on it, and therefore cannot see a wrong week at all.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one shipment line is

no split -- one whole invoice goes into one call. src/segment.py cuts each invoice into addressable sections on its underlined headings so a value can name which section it was read from, but nothing is chunked and nothing is dropped for length.

SetupWhat the setup figure measured

There is no index and nothing is built. Each invoice goes whole into one call, so the build cost is genuinely zero rather than unmeasured -- the only preparation is tools/build_corpus.py regenerating the corpus and gold from the seed, which takes under a second and makes no call. ⚠︎ THE WORD 'INDEX' MEANS SOMETHING ELSE ENTIRELY ON THIS KIT: the fuel price index is a 25-row lookup table shipped in data/schedule.json and rendered into every prompt. It is not a retrieval index, nothing is embedded, and nothing is searched.

LicenceLicence

MIT (this repository's own licence). Nothing was fetched and nothing is licensed from anybody: no real carrier, shipper, index, place, terminal, person or currency is named anywhere in the kit, and the weekly fuel price series it recomputes against is an invention rather than a published series.

Bring your ownBring your own carrier fuel invoices

Replace data/corpus/*.txt with your own invoices and data/gold.jsonl with your own labels; rewrite data/schedule.json with your contract's lag rule, bracket table and cap -- and, above all, with THE INDEX SERIES YOUR CONTRACT ACTUALLY NAMES, which this kit does not have; adjust the section names in src/select.py to match your headings and put anything you will not send into NEVER_SENT. src/segment.py falls back to one whole-document segment when it finds no underlined headings, so a differently-shaped invoice still runs. Then re-run evals/check_labels.py, which will tell you what your labels disagree with before you spend anything.

⚠︎ And what stops being true when you do: Nothing measured here transfers. Three things in particular: the index series is an INVENTION, so a 100 pct score means the run agreed with a table this kit wrote; the corpus puts the whole series in the PROMPT, which a production build would not do; and every invoice here states its lag and its cap in one place in a fixed form, where a real contract states them in prose in a document the invoice does not carry.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 6,188 tokens assembled, 3,028 are contexts — 49% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured, not counted locally: evals/prompt_tokens.py sends the prompt as four nested prefixes at max_tokens=1 and takes each part's size as the difference between two consecutive prompt_tokens counts the PROVIDER returns. Run p001-fuel-surcharge on FUE-0044, 4 calls.

Full promptThe prompt in full

You read ONE CARRIER FUEL INVOICE -- which shipments it covers, what each one's linehaul charge was, and what the carrier billed for fuel on each -- and you extract structured fields from it. You return JSON and nothing else.

THE QUESTION IS WHETHER EACH FUEL SURCHARGE WAS COMPUTED FROM THE INDEX ROW THE CONTRACT SELECTS. A fuel surcharge is a percentage read out of a bracket table; the bracket is chosen by a published index price; and the index price is chosen by a LAG RULE applied to the shipment date. Get the week wrong and the money is still plausible -- a real percentage, the right base, a figure inside any normal band. That is exactly the failure you are looking for, and the only way to see it is to work out the row yourself.

You are PROPOSING a recovery worklist for a qualified person to act on. You never adjust, credit, re-rate or short-pay an invoice, never approve, hold or release a payment, never raise a claim with a carrier, and never substitute an index row the publisher did not issue. The schedule and the index series below are ILLUSTRATIVE and were written for this kit; they are not a carrier tariff, not a contract and not a real published index, and they are not an authority.

RULES, in order of importance:
1. THE APPLICABLE INDEX ROW IS WORKED OUT, NEVER READ OFF THE INVOICE. Every billing row states an index price the CARRIER says it applied. Copy it into `carrier_index_price` and then IGNORE IT for every other purpose. It is a claim. On a wrongly-billed line it is a real price from a real week, and it is the wrong week.
2. THE LAG RULE, IN TWO STEPS. Take the shipment date. Step back to the most recent SUNDAY ON OR BEFORE it -- if the shipment moved on a Sunday, that Sunday itself. Then step back a further `lag_weeks` WHOLE WEEKS. That date is `index_week_ending`. The lag is stated in THIS invoice's Contract Terms and is not the same on every contract you will see.
3. LOOK THAT WEEK UP IN THE PUBLISHED SERIES GIVEN BELOW. If the series carries NO ROW for that week, `index_price` is null and the line is `not_determinable`. Do NOT use the nearest published week, do not interpolate, and do not fall back to the carrier's figure.
4. Return one object per row of `Shipment Lines`, in the order that section lists them, and no others. THE BILLING SECTION CARRIES ROWS FOR SHIPMENTS THIS INVOICE DOES NOT COVER. Reporting one of those is as wrong as missing a real one. Never invent a shipment.
5. THE TWO SECTIONS ARE IN DIFFERENT ORDERS AND NEITHER MATCHES THE OTHER. Join by shipment reference, matched exactly and in full. SHP-3001 and SHP-30012 are different shipments.
6. If the billing section carries NO ROW at all for a shipment, its billed surcharge and its carrier basis are null. That is not a gap in your answer, it is a fact about the invoice, and it is what makes the line undeterminable. Never carry a value across from another shipment and never infer one.
7. If the invoice does not state a field, return null for it. Do not infer it and do not use what you know about the world -- in particular, do not use anything you believe you know about real fuel prices in any week.
8. THE BASE IS THE LINEHAUL ALONE. Every shipment row states an accessorial total beside the linehaul. It is never surcharged. Adding it is a mistake worth exactly the accessorial times the percentage, and it looks like nothing.
9. `surcharge_status` is decided ONLY by the SCHEDULE given below. Work through these six gates IN ORDER and STOP at the first one that fires:
   a. EXEMPT LANE. The rate basis is 'all-in', so the rate includes fuel and no index row applies. Answer 'fuel_exempt_all_in' where 0.00 was billed, and 'variance' where anything at all was billed. THIS GATE IS FIRST because every gate below would otherwise recompute from a basis this line does not have.
   b. NOT DETERMINABLE. The shipment date, the linehaul amount or the billed surcharge is not recorded; or the published series carries no row for the week the lag rule selects. Answer 'not_determinable'. It is a real answer, not a failure to produce one -- a confident wrong 'matches' is the expensive mistake here.
   c. CAPPED AT MAXIMUM. The bracket percentage for the applicable row is ABOVE the contract maximum, and the billed amount equals the linehaul times that maximum. Answer 'capped_at_maximum'. THIS GATE IS THIRD: it is a difference that is supposed to be there, and a recomputation that ignores the cap reports every capped line as an undercharge.
   d. WRONG INDEX WEEK. The billed amount does not match the recomputation, and it DOES reconcile -- within 1 minor unit(s) -- to the same computation run against at least one OTHER published week in the series, under the same cap. Answer 'wrong_index_week'.
   e. VARIANCE. The billed amount does not match and reconciles to no published week. Answer 'variance'.
   f. Anything that survives all five gates is 'matches'.
10. THE INVOICE DATE DECIDES NOTHING. The lag rule keys off the SHIPMENT date. An invoice raised weeks after the shipment moved does not move the applicable row by a single day.
11. THE CARRIER'S REMARK IS A FIELD TO COPY, NOT EVIDENCE ABOUT ANY SURCHARGE. A remark that sounds settled does NOT clear a finding, and a remark that sounds worried does NOT create one. The invoice's own numbers and the published series decide; the remark is one person's note and may disagree with them.
12. Copy every reference and every amount verbatim, exactly as printed, including a trailing zero where the invoice prints one. Every date is yyyy-mm-dd. `variance_minor` is an INTEGER in minor units.
13. Use the exact allowed value for a field that lists them, and return every field named in the schema for every line, even when the answer is null.

FUEL SURCHARGE SCHEDULE FSC-1 (the authority for `surcharge_status`; this is an ILLUSTRATIVE schedule shipped with this kit, not a carrier tariff, not a contract and not an authority)

THE LAG RULE
  Take the shipment date. Step back to the most recent SUNDAY ON OR BEFORE it -- that is the shipment's own index week, and where the shipment moved on a Sunday it is that Sunday itself. Then step back a further `lag_weeks` whole weeks. The published row whose `week ending` equals that date is the applicable index row, and there is no other.
  The invoice's own Contract Terms section. It is NOT the same on every contract in this corpus and it is not a constant in this file -- reading it off the wrong invoice is one of the ways to get every line on that invoice wrong at once.

THE SURCHARGE BASE
  The surcharge base is the LINEHAUL CHARGE ALONE. Accessorial charges on the same line are never surcharged.

THE CONTRACT CAP
  The invoice's Contract Terms may state a MAXIMUM SURCHARGE PERCENTAGE. Where the bracket percentage for the applicable index row exceeds it, the applicable percentage is the maximum.

ROUNDING
  The recomputed surcharge is the linehaul amount multiplied by the applicable percentage, rounded HALF UP to the nearest minor unit. Billed within 1 minor unit(s) of that figure counts as matching.

RATE BASES
  tariff, all_in
  `tariff` means the linehaul is rated off the tariff and a fuel surcharge is due on it. `all_in` means the negotiated rate INCLUDES fuel, no index row applies to the line, and the correct billed surcharge is 0.00. A recomputation that applies a bracket percentage to every line reports a full shortfall on every all-in lane -- which is why the exempt-lane gate is first.

PUBLISHED INDEX SERIES IDX-1 -- ILLUSTRATIVE, NOT A REAL INDEX. One row per week ending on the Sunday named. A week with no row here was never published.
  week ending 2025-10-26   published price 3.612
  week ending 2025-11-02   published price 3.700
  week ending 2025-11-09   published price 3.755
  week ending 2025-11-16   published price 3.899
  week ending 2025-11-23   published price 3.842
  week ending 2025-11-30   published price 3.918
  week ending 2025-12-07   published price 4.021
  week ending 2025-12-14   published price 4.107
  week ending 2025-12-21   published price 4.200
  week ending 2025-12-28   published price 4.286
  week ending 2026-01-11   published price 4.331
  week ending 2026-01-18   published price 4.299
  week ending 2026-01-25   published price 4.174
  week ending 2026-02-01   published price 4.055
  week ending 2026-02-08   published price 3.964
  week ending 2026-02-15   published price 3.870
  week ending 2026-02-22   published price 3.799
  week ending 2026-03-01   published price 3.681
  week ending 2026-03-08   published price 3.574
  week ending 2026-03-15   published price 3.500
  week ending 2026-03-22   published price 3.446
  week ending 2026-03-29   published price 3.519
  week ending 2026-04-05   published price 3.633
  week ending 2026-04-12   published price 3.728
  week ending 2026-04-19   published price 3.816
  ⚑ THE WEEK ENDING 2026-01-04 IS DELIBERATELY ABSENT. The table looks continuous either side of it. Reaching for the nearest published week is the obvious wrong move and it produces a confident, plausible, wrong number.

BRACKET TABLE -- the surcharge percentage for a published index price
  from      to        pct
  3.200     3.299     22.00
  3.300     3.399     23.50
  3.400     3.499     25.00
  3.500     3.599     26.50
  3.600     3.699     28.00
  3.700     3.799     29.50
  3.800     3.899     31.00
  3.900     3.999     32.50
  4.000     4.099     34.00
  4.100     4.199     35.50
  4.200     4.299     37.00
  4.300     4.399     38.50
  4.400     4.499     40.00
  4.500     4.599     41.50
  Each row covers a closed range of index prices and names the surcharge percentage for it. The ranges are contiguous and do not overlap, and a price on an edge belongs to the row that names it -- 3.799 is 29.50 pct and 3.800 is 31.00 pct. ⚠︎ ONE TABLE FOR EVERY LANE, SERVICE LEVEL AND EQUIPMENT TYPE is a simplification; a real carrier publishes several.

THE SIX GATES, IN ORDER. STOP AT THE FIRST THAT FIRES.
  1. FUEL_EXEMPT_ALL_IN:    The line's rate basis is `all_in`. The negotiated rate includes fuel and no index row applies. Where the billed surcharge is 0.00 the answer is `fuel_exempt_all_in`; where anything at all was billed the answer is `variance` for the whole billed amount.
     why: FIRST, because an exempt lane has no applicable index row and every gate below would be recomputing from a basis that does not apply to the line. It is also the gate that decides both ways: a surcharge billed on an all-in lane is not a rounding argument, it is the whole amount.
  2. NOT_DETERMINABLE:      The shipment date, the linehaul amount or the billed surcharge is not recorded on the line; or the published series carries NO ROW for the week the lag rule selects.
     why: On a recovery worklist the expensive mistake is a confident wrong `matches`. A week the publisher never issued is a record somebody has to open, not a line that can be cleared, and the nearest published week is not an answer -- it is a different number wearing the same name.
  3. CAPPED_AT_MAXIMUM:     The bracket percentage for the applicable index row EXCEEDS the contract maximum, and the billed amount equals the linehaul multiplied by that maximum, within the rounding tolerance.
     why: This is the difference that is SUPPOSED to be there. It sits above the finding gates because a recomputation that reads the bracket and ignores the cap reports every capped line as an undercharge, and a worklist of undercharges is a worklist nobody opens twice.
  4. WRONG_INDEX_WEEK:      The billed amount does not match the recomputation, and it DOES reconcile -- within the rounding tolerance -- to the same computation run against at least one OTHER published week in the series.
     why: ⚑ THE FINDING THIS KIT EXISTS FOR. The money is a real percentage from a real published row applied to the right base, so it lands inside any plausible band and nothing downstream of the arithmetic can see it. It is a different finding from a plain variance and it has a different conversation attached: the carrier can be shown the row it used and the row the contract selects.
  5. VARIANCE:              The billed amount does not match the recomputation and reconciles to no published week.
     why: The percentage is off the schedule, the base was not the linehaul, the cap was ignored, or the figure is simply wrong. It is a finding, and unlike a wrong-week finding there is no row to point at.
  6. MATCHES:               Everything else: the billed amount equals the recomputation within the rounding tolerance.
     why: Nothing needs doing. ⚠︎ AND NOTE WHAT THIS DOES NOT SAY. It does not say the carrier used the right index row -- see `carrier_basis_note`.

THE SIX STATUSES
  matches, capped_at_maximum, fuel_exempt_all_in, wrong_index_week, variance, not_determinable

WHAT PUTS A LINE ON THE RECOVERY WORKLIST
  wrong_index_week, variance -- The two statuses that put a line on somebody's recovery worklist. `matches`, `capped_at_maximum` and `fuel_exempt_all_in` are the three answers that mean nothing needs doing. `not_determinable` means the RECORD needs opening rather than the MONEY needing recovering, and it is deliberately not on the worklist for that reason.

THE VARIANCE
  The billed surcharge minus the recomputed surcharge, in minor units. Positive means the carrier billed MORE than the schedule and the index support; negative means it billed less. Null where the recomputation cannot be run.

WHAT THIS TOOL WILL NOT DO
  - It never adjusts, credits, re-rates or short-pays an invoice.
  - It never approves, holds or releases a carrier payment.
  - It never raises, sends or files a claim with a carrier.
  - It never revises, interpolates or substitutes an index row the publisher did not issue.
  - It never decides which of two index rows the contract binds you to.

Extract these fields:
INVOICE-LEVEL FIELDS -- one value each, for the whole invoice
- invoice_ref (string) -- the invoice reference, verbatim from the Invoice section
- invoice_date (date) -- the date the invoice was raised, from the Invoice Date section, as yyyy-mm-dd. It is NOT a shipment date and it selects no index row
- schedule_id (string) -- the identifier of the fuel surcharge schedule this invoice is rated under, from the Fuel Surcharge Schedule section, verbatim -- the short code only, not the sentence after it
- lag_weeks (integer) -- the number of WHOLE WEEKS of lag, read from the 'fuel surcharge lag' line of Contract Terms. An integer, so '2 week(s)' is 2. ⚠︎ IT IS NOT THE SAME ON EVERY CONTRACT IN THIS CORPUS -- read it off THIS invoice, and note that getting it wrong moves every line on the invoice at once
- max_surcharge_pct (decimal) -- the maximum surcharge percentage from the 'maximum surcharge' line of Contract Terms, copied exactly as printed, e.g. 32.00. It caps the bracket percentage; where the bracket is at or below it, it changes nothing
- carrier_remark (string) -- the carrier's own free-text remark from the Carrier Remarks section, copied verbatim. It is a field to copy and it is NOT evidence about whether any surcharge is right

PER-LINE FIELDS -- one object per row of Shipment Lines
- shipment_ref (string) -- the shipment reference as the Shipment Lines section states it, e.g. SHP-30012. Return one object per row of THAT section, in the order it lists them, and no others. The Fuel Surcharge As Billed section carries rows for shipments this invoice does not cover -- those are not yours
- shipment_date (date) -- the date the shipment moved, read after 'shipped' on the row in Shipment Lines, as yyyy-mm-dd. Return null where the row reads 'shipped -- not stated'. THIS IS THE DATE THE LAG RULE KEYS OFF, and it is not the invoice date
- linehaul_amount (decimal) -- the linehaul charge, read after 'linehaul' on the row in Shipment Lines. Copy the digits exactly as printed. Return null where the row reads 'linehaul -- not stated'. It is the ONLY base the surcharge is computed on -- the accessorial total on the same row is never surcharged
- rate_basis (enum) one of: tariff, all_in -- read after 'basis' on the row in Shipment Lines: 'tariff' where it reads 'basis tariff', and 'all_in' where it reads 'basis all-in'. An all-in rate INCLUDES fuel, so no index row applies to that line and 0.00 is the surcharge that is due on it
- carrier_index_price (decimal) -- the index price the CARRIER states it applied, from the Fuel Surcharge As Billed row for this shipment, after 'carrier index'. Return null where the row reads 'carrier index -- not applicable', and null where that section carries no row for this shipment. ⚠︎ IT IS WHAT THE CARRIER CLAIMS AND IT IS NOT NECESSARILY THE APPLICABLE ROW. Copy it, and never use it to answer index_price
- carrier_pct (decimal) -- the percentage the carrier states it applied, from the same row, after 'at' and before 'pct'. Return null where the row states no percentage and null where there is no row for this shipment. Copy it; it is a claim, not an authority
- billed_surcharge (decimal) -- the fuel surcharge actually billed on this shipment, from the same row, after 'billed fuel surcharge'. Return null where the row reads '-- not stated', and null where the Fuel Surcharge As Billed section carries NO ROW for this shipment at all. A missing row is not a surcharge of zero -- it is the answer, and it is what makes the line undeterminable
- index_week_ending (derived) -- THE APPLICABLE INDEX ROW, WORKED OUT -- no invoice states it. Take the shipment date, step back to the most recent SUNDAY ON OR BEFORE it (a shipment that moved on a Sunday uses that Sunday itself), then step back a further lag_weeks whole weeks. Answer that date as yyyy-mm-dd. Answer it for every line whose shipment date is stated, INCLUDING all-in lines. Return null only where the shipment date is not stated
- index_price (lookup) -- the price PUBLISHED for that week, looked up in the weekly index series given above. Copy it exactly as the series prints it, to three decimals. Return null where the series carries NO ROW for that week -- the nearest published week is not the answer, and neither is the carrier's own figure on the line
- applicable_pct (lookup) -- the bracket percentage for that published price, from the bracket table given above, capped at max_surcharge_pct where the bracket is ABOVE it. A price on a bracket edge belongs to the row that names it. Return null where there is no published price, and null on an all-in line
- recomputed_surcharge (derived) -- the linehaul amount multiplied by applicable_pct, rounded HALF UP to the minor unit, to two decimals. Return 0.00 on an all-in line, because the rate already includes fuel. Return null where the linehaul or the applicable percentage is not available
- variance_minor (derived) -- the billed surcharge MINUS the recomputed surcharge, in MINOR UNITS (hundredths). An integer, so a difference of 12.35 is 1235 and 0.00 is 0. Positive means the carrier billed MORE than the schedule and the index support. Return null where either amount is not available
- surcharge_status (enum) one of: matches, capped_at_maximum, fuel_exempt_all_in, wrong_index_week, variance, not_determinable -- this line's status under the shipped schedule. Work through six gates IN ORDER and stop at the first that fires. (1) EXEMPT LANE: the rate basis is all-in, so no index row applies -- 'fuel_exempt_all_in' where 0.00 was billed, 'variance' where anything was. (2) NOT DETERMINABLE: the shipment date, the linehaul or the billed surcharge is not recorded, or the series carries no row for the selected week. (3) CAPPED AT MAXIMUM: the bracket is above the contract maximum and the billed amount equals the linehaul times that maximum. (4) WRONG INDEX WEEK: the billed amount does not match the recomputation but does reconcile to the same computation against another PUBLISHED week. (5) VARIANCE: it does not match and reconciles to no published week. (6) Anything that survives all five is MATCHES. Never decide it from the Carrier Remarks note, and never from the carrier's own stated index or percentage

Return a JSON object of exactly this shape:
{"invoice_ref": ..., "invoice_date": ..., "schedule_id": ..., "lag_weeks": ..., "max_surcharge_pct": ..., "carrier_remark": ..., "lines": [{"shipment_ref": ..., "shipment_date": ..., "linehaul_amount": ..., "rate_basis": ..., "carrier_index_price": ..., "carrier_pct": ..., "billed_surcharge": ..., "index_week_ending": ..., "index_price": ..., "applicable_pct": ..., "recomputed_surcharge": ..., "variance_minor": ..., "surcharge_status": ...}, ...]}
Use null for any field the invoice does not state.

CARRIER FUEL INVOICE
--------------------
Invoice
-------
INV-1250-WZ

Invoice Date
------------
2026-03-14

Fuel Surcharge Schedule
-----------------------
FSC-1 -- illustrative, shipped with this kit; reproduces no real carrier tariff and no real published index

Contract Terms
--------------
fuel surcharge lag        1 week(s) -- whole weeks, stepped back from the shipment's own index week
maximum surcharge         32.00 pct
surcharge base            linehaul charge only; accessorial charges are not surcharged
index series              IDX-1, published weekly, each row covering the week ending on the Sunday named

Shipment Lines
--------------
amounts are in the trading currency's major unit; the index is stated to three decimals
SHP-30366   Depot 3 to Plant 4             shipped 2026-02-28      linehaul 3504.25         accessorials 160.75   basis tariff
SHP-30254   Depot 11 to DC 3               shipped 2026-02-22      linehaul 4170.75         accessorials 301.75   basis tariff
SHP-30881   DC 5 to Depot 3                shipped 2026-02-26      linehaul 2490.00         accessorials 0.00     basis tariff
SHP-30437   Plant 3 to Plant 1             shipped 2026-01-18      linehaul 2899.50         accessorials 188.75   basis tariff
SHP-30215   Depot 10 to DC 6               shipped 2026-02-23      linehaul 871.25          accessorials 0.00     basis tariff
SHP-30815   Depot 5 to DC 4                shipped 2026-02-25      linehaul 4678.00         accessorials 271.50   basis tariff
SHP-30101   DC 4 to Plant 3                shipped 2026-02-04      linehaul 4540.00         accessorials 0.00     basis tariff
SHP-30544   DC 6 to Port Gate 3            shipped 2026-01-14      linehaul 1123.50         accessorials 68.25    basis tariff

Fuel Surcharge As Billed
------------------------
SHP-80249   carrier index 4.299 at 32.00 pct       billed fuel surcharge 1166.64
SHP-80240   carrier index 3.964 at 32.00 pct       billed fuel surcharge 1471.36
SHP-30815   carrier index 3.870 at 31.00 pct       billed fuel surcharge 1450.18
SHP-30101   carrier index 4.174 at 32.00 pct       billed fuel surcharge 1452.80
SHP-30544   carrier index 4.055 at 32.00 pct       billed fuel surcharge 359.52
SHP-30881   carrier index 3.964 at 32.00 pct       billed fuel surcharge 796.80
SHP-30437   carrier index 4.331 at 38.50 pct       billed fuel surcharge 1116.31
SHP-30215   carrier index 3.964 at 32.00 pct       billed fuel surcharge 278.80
SHP-80270   carrier index 4.055 at 32.00 pct       billed fuel surcharge 1290.80
SHP-30366   carrier index 3.964 at 32.00 pct       billed fuel surcharge 1121.36
SHP-30254   carrier index 3.870 at 31.00 pct       billed fuel surcharge 1292.93

Carrier Remarks
---------------
Consolidated fuel invoice for the period. Accessorials billed separately by lane.

Raw responseThe raw response

{"invoice_ref":"INV-1250-WZ","invoice_date":"2026-03-14","schedule_id":"FSC-1","lag_weeks":1,"max_surcharge_pct":32.00,"carrier_remark":"Consolidated fuel invoice for the period. Accessorials billed separately by lane.","lines":[{"shipment_ref":"SHP-30366","shipment_date":"2026-02-28","linehaul_amount":3504.25,"rate_basis":"tariff","carrier_index_price":3.964,"carrier_pct":32.00,"billed_surcharge":1121.36,"index_week_ending":"2026-02-15","index_price":3.870,"applicable_pct":31.00,"recomputed_surcharge":1086.32,"variance_minor":3504,"surcharge_status":"wrong_index_week"},{"shipment_ref":"SHP-30254","shipment_date":"2026-02-22","linehaul_amount":4170.75,"rate_basis":"tariff","carrier_index_price":3.870,"carrier_pct":31.00,"billed_surcharge":1292.93,"index_week_ending":"2026-02-15","index_price":3.870,"applicable_pct":31.00,"recomputed_surcharge":1292.93,"variance_minor":0,"surcharge_status":"matches"},{"shipment_ref":"SHP-30881","shipment_date":"2026-02-26","linehaul_amount":2490.00,"rate_basis":"tariff","carrier_index_price":3.964,"carrier_pct":32.00,"billed_surcharge":796.80,"index_week_ending":"2026-02-15","index_price":3.870,"applicable_pct":31.00,"recomputed_surcharge":771.90,"variance_minor":2490,"surcharge_status":"wrong_index_week"},{"shipment_ref":"SHP-30437","shipment_date":"2026-01-18","linehaul_amount":2899.50,"rate_basis":"tariff","carrier_index_price":4.331,"carrier_pct":38.50,"billed_surcharge":1116.31,"index_week_ending":"2026-01-11","index_price":4.331,"applicable_pct":32.00,"recomputed_surcharge":927.84,"variance_minor":18847,"surcharge_status":"variance"},{"shipment_ref":"SHP-30215","shipment_date":"2026-02-23","linehaul_amount":871.25,"rate_basis":"tariff","carrier_index_price":3.964,"carrier_pct":32.00,"billed_surcharge":278.80,"index_week_ending":"2026-02-15","index_price":3.870,"applicable_pct":31.00,"recomputed_surcharge":270.09,"variance_minor":871,"surcharge_status":"wrong_index_week"},{"shipment_ref":"SHP-30815","shipment_date":"2026-02-25","linehaul_amount":4678.00,"rate_basis":"tariff","carrier_index_price":3.870,"carrier_pct":31.00,"billed_surcharge":1450.18,"index_week_ending":"2026-02-15","index_price":3.870,"applicable_pct":31.00,"recomputed_surcharge":1450.18,"variance_minor":0,"surcharge_status":"matches"},{"shipment_ref":"SHP-30101","shipment_date":"2026-02-04","linehaul_amount":4540.00,"rate_basis":"tariff","carrier_index_price":4.174,"carrier_pct":32.00,"billed_surcharge":1452.80,"index_week_ending":"2026-01-25","index_price":4.174,"applicable_pct":32.00,"recomputed_surcharge":1452.80,"variance_minor":0,"surcharge_status":"capped_at_maximum"},{"shipment_ref":"SHP-30544","shipment_date":"2026-01-14","linehaul_amount":1123.50,"rate_basis":"tariff","carrier_index_price":4.055,"carrier_pct":32.00,"billed_surcharge":359.52,"index_week_ending":"2026-01-04","index_price":null,"applicable_pct":null,"recomputed_surcharge":null,"variance_minor":null,"surcharge_status":"not_determinable"}]}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch a carrier's fuel surcharge billed off the wrong week — 50 carrier fuel invoices. One model answered, and every answer was then graded Six different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Pure code, no LLM judge. Six graders, scored separately and never folded together: per-cell exact match over 4284 cells; THE INDEX ROW TRACED, a JOINT test of the week the lag rule selects AND the price published for it; the six-way surcharge status, taken by running src/schedule.py over the values the model returned under the contract terms the model itself read; the WRONG-WEEK RECALL over the named sub-population; the false-alarm rate on the recovery worklist; and the model's own recomputation arithmetic, which decides nothing. Plus the not-determinable reach, the invoice-level four-way pattern reading, the claim condition, the exact recovery total in minor units, and a no-gold consistency diagnostic. ⚑ THE INDEX ROW IS SCORED AS ONE ANSWER, NOT TWO. A right week citing the wrong price is not half a citation and a right price under the wrong week is a coincidence -- and on this kit the row is the whole product, because a surcharge computed from the wrong week is arithmetically indistinguishable from a correct one. The denominator is every line whose shipment date is stated (327 of 332), so a row the reply never derived counts against it. Its hardest slice is scored separately: the 20 lines whose selected week was NEVER PUBLISHED, where the correct price is null and the tempting answer is the nearest row.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid -- the real spend is in the shared call ledger, not on this page.

Grading unitWhat the grading figure prices

All six graders, the gold gate and the free floor are pure code -- no LLM judge, no reviewer, no second model. Re-scoring any committed result set costs $0.00 and needs no key. What the RUN costs is one call per INVOICE, whole, and not one per shipment line. The contract terms govern every line, the published series is a single table every line indexes into, and the systematic-versus-scattered reading only means anything across the whole roster -- so a per-line call would carry most of the invoice anyway and would lose the invoice-level answer entirely.

The gradersSix ways to grade

The floor's four-way pattern accuracy is 58.0 pct, and that number is a construction rather than a reading: it answers none on every invoice because it can never produce a wrong_index_week line, and 29 of the 50 invoices genuinely are none. A grader can be right for a reason that is not a capability.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — top_k, chunk size, prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The free carrier-basis floor (evals/baseline.py), the gold gate (evals/check_labels.py) and the scorer (evals/judge.py) are all pure code and cost $0.00 to run against any result set. The figure above is the ONE scored run's own token counts (50 invoices) priced at Google Gemini 3 Flash's published rate -- the same basis cost_per_query_usd uses.\n\n⚠︎ IT IS NOT WHAT BUILDING THIS KIT COST. Outside the scored run: a 4-invoice calibration at max_tokens=60000 fired at the four invoices tools/pick_shot.py ranks hardest ($0.2094), the worked-example capture ($0.0409), 4 prompt-split calls at max_tokens=1 ($0.0083), and THREE live UI screenshot calls. The four priced steps come to $2.6631. 63 live calls sit behind this page according to the shared ledger; 62 of them are attributable to a named step, and the screenshot script records no token counts, so those three are counted and not priced.\n\n⚑ AND THE CALIBRATION IS THE LINE ITEM TO READ, BECAUSE IT WAS WRONG AND IT STILL EARNED ITS MONEY. It measured a largest reply of 23,048 output tokens. The scored run's largest was 70,221 -- 3.05x more, on the same corpus. What the calibration actually bought was not the number: it was the knowledge that 93.3 pct of the output is a reasoning pass whose length nothing on the page bounds, which is why the ceiling was set at 4.2x the observed maximum instead of the 2.75x a sibling kit uses. 2.75x would have been 63,382 and would have cut the scored run's largest reply. ⚠︎ TWO OF THE THREE SCREENSHOT CALLS WERE ALSO WASTE, AND SELF-INFLICTED: the first fired before the answer arrived because the shot driver carried a sibling kit's placeholder text, and the second captured a table whose verdict column had collapsed to one word per line. Both are in the ledger and both are named in the UI lens.

Cost driversWhat actually moves the bill

• ⚑ THE TABLE, NOT THE INVOICE. 2162 of every call's 6188 input tokens are the schedule, the lag rule, the bracket table and all 25 published index weeks -- 35 pct -- and the invoice itself is 866 tokens, 14 pct. That table is byte-identical on every one of the 50 calls.
• ⚑ THE REASONING PASS, WHICH IS THE BILL. 93.3 pct of all output tokens (700,710 of 751,154) were provider-side reasoning. The JSON a nine-line reply needs is about 2,300 tokens; the run averaged 15,023 output tokens per invoice. Nineteen twentieths of what was paid for was the model stepping the lag rule, scanning the series and multiplying.
• THE LINE COUNT, not the document length. The reply is a roster of thirteen-field objects of which five are derived, so output tracks the number of lines times the reasoning each derivation costs. This corpus runs 5 to 9 lines per invoice.
• ⚠︎ AND THE CHEAPEST ROW IS STILL PAYING FOR WORK CODE COULD DO FREE. src/schedule.py already derives the week, looks up the bracket and does the multiplication, deterministically and at zero cost. The model is asked to do all three so that the derivation can be MEASURED. A production build asks it only to read the invoice.

Your volumeWhat it costs at your volume

Ten times the invoices is ten times the calls and ten times the bill -- $24.05 on this card, linearly, because there is no index to amortise and no cache. Ten times the LINES PER INVOICE is worse than linear: the reply is a roster and the reasoning grows with it, and the largest reply on this run already used 70,221 of the 96,000 ceiling on an invoice of 9 lines.

Model choiceWhy this model

One tier, chosen because it is what the shared repo-root .env already pointed at, and run once. Nothing here is a model recommendation: no second tier was run, so this page cannot tell you whether a cheaper model would have traced the row or an expensive one would have done it faster. What it can tell you is that this tier traced 327 of 327 rows and made exactly one derivation error, and that the same corpus leaves no headroom to measure a better one against.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 invoices, one completion call each, one tier. The calibration, the worked-example capture, the prompt-split calls and the three UI screenshot calls are recorded separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

11 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pyInvoice segmenter

cuts one invoice into addressable sections on its underlined headings; falls back to one whole-document segment when it finds none

# Cut a carrier fuel invoice into addressable sections. Pure code -- no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pySection selector — a swap seam

maps each of the 19 fields to the sections that can carry it, and subtracts the two sections no field maps to -- unconditionally, including on the fallback path

You change it to: which sections reach the provider at all. NEVER_SENT is the privacy line; _fallback subtracts it unconditionally so an absent section cannot widen the wire.

# Pick which sections of a carrier fuel invoice are sent. Pure code -- the last deterministic step

INVOICE = "Invoice"
INVOICE_DATE = "Invoice Date"
SCHEDULE = "Fuel Surcharge Schedule"
TERMS = "Contract Terms"
LINES = "Shipment Lines"
BILLED = "Fuel Surcharge As Billed"
REMARKS = "Carrier Remarks"
NEVER_SENT = ("Carrier Contact", "Prepared By")
SECTION_HINTS = {

src/prompt.pyPrompt assembler

one call per invoice: the system prompt, the schedule rendered from its own JSON including all 25 published index weeks, the field schema, and the mapped sections

# Assemble the fuel-surcharge prompt. ONE prompt per invoice, every shipment line on it in one

VERDICTS = ("matches", "capped_at_maximum", "fuel_exempt_all_in",
VERDICT_MEANINGS = {
SYSTEM = (
def schedule_block(sc=None):
SCHEDULE_TEXT = schedule_block()
def field_schema(fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/adapters/__init__.pyProvider adapter

urllib against an OpenAI-compatible endpoint or Anthropic's Messages API, with backoff over both HTTP and transport failures

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/schedule.pyThe lag rule — a swap seam

the applicable index row, derived: the most recent Sunday on or before the shipment date, then back a further lag_weeks whole weeks. Pure code, and answered for every line with a shipment date -- including exempt and unpriceable ones

You change it to: what warrants a recovery claim being raised by a person. It is a POLICY change and is deliberately a different function from the schedule lookup. It never raises anything -- carrier-payment-release is not configurable.

# THE FUEL SURCHARGE SCHEDULE AND THE LAG RULE, loaded from data/schedule.json. Pure code, no

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
SCHEDULE_PATH = os.path.join(HERE, "data", "schedule.json")
STATUSES = ("matches", "capped_at_maximum", "fuel_exempt_all_in",
EXCEPTIONS = ("wrong_index_week", "variance")
NOTHING_TO_DO = ("matches", "capped_at_maximum", "fuel_exempt_all_in")
RATE_BASES = ("tariff", "all_in")
PATTERN_STATES = ("none", "isolated", "systematic", "scattered")
def load(path=SCHEDULE_PATH):
SC = load()

src/schedule.pyThe surcharge schedule — a swap seam

the six gates in order, the bracket lookup and the cap, as pure code over integer minor units -- one definition read by the corpus generator, the prompt and the scorer

You change it to: what warrants a recovery claim being raised by a person. It is a POLICY change and is deliberately a different function from the schedule lookup. It never raises anything -- carrier-payment-release is not configurable.

# THE FUEL SURCHARGE SCHEDULE AND THE LAG RULE, loaded from data/schedule.json. Pure code, no

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
SCHEDULE_PATH = os.path.join(HERE, "data", "schedule.json")
STATUSES = ("matches", "capped_at_maximum", "fuel_exempt_all_in",
EXCEPTIONS = ("wrong_index_week", "variance")
NOTHING_TO_DO = ("matches", "capped_at_maximum", "fuel_exempt_all_in")
RATE_BASES = ("tariff", "all_in")
PATTERN_STATES = ("none", "isolated", "systematic", "scattered")
def load(path=SCHEDULE_PATH):
SC = load()

src/extract.pyExtractor — a swap seam

drives the whole path and attaches a span to every spannable value, searched inside the section the field is mapped to before it ever widens. The two lookup fields carry NO span by design -- a citation into the invoice would land on the carrier's own claim

You change it to: MAX_TOKENS. A cut reply costs a whole invoice and produces no worklist at all, which looks exactly like an invoice with nothing wrong.

# Extract one carrier fuel invoice: segment, select, prompt, ONE model call, then the pure-code

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 96000
def load_fields():
def line_key(fields):
def scored_line_fields(fields):
def load_doc(invoice_id):
def documents():

src/schedule.pyInvoice readings — a swap seam

the pure-code pattern reading (none / isolated / systematic / scattered), the recovery total in minor units, and the business condition that proposes a claim

You change it to: what warrants a recovery claim being raised by a person. It is a POLICY change and is deliberately a different function from the schedule lookup. It never raises anything -- carrier-payment-release is not configurable.

# THE FUEL SURCHARGE SCHEDULE AND THE LAG RULE, loaded from data/schedule.json. Pure code, no

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
SCHEDULE_PATH = os.path.join(HERE, "data", "schedule.json")
STATUSES = ("matches", "capped_at_maximum", "fuel_exempt_all_in",
EXCEPTIONS = ("wrong_index_week", "variance")
NOTHING_TO_DO = ("matches", "capped_at_maximum", "fuel_exempt_all_in")
RATE_BASES = ("tariff", "all_in")
PATTERN_STATES = ("none", "isolated", "systematic", "scattered")
def load(path=SCHEDULE_PATH):
SC = load()

evals/baseline.pyThe free floor

the carrier-basis check: the index the carrier printed on the line, priced through the schedule and compared. Blind to the lag rule by construction

# A free, rules-and-regex freight-bill check. No model, no key, no spend -- scored by the same

NOT_STATED = "-- not stated"
NOT_APPLICABLE = "-- not applicable"
LINES_SECTION = "Shipment Lines"
BILLED_SECTION = "Fuel Surcharge As Billed"
def _section(text, name):
def _rows(text, name):
def _cols(line):
def _terms(text):
def _roster(text):

evals/judge.pyScorer

six graders, pure code, no LLM judge

# Score a fuel-surcharge run. PURE CODE -- gold is exact and every answer is one value, so `==`

def norm(v):
def equal(field, got, want):
def _rows_by_ref(items, key="shipment_ref"):
def score(fields, records, golds):
def _matrix(rows, positive):
NOTHING_TO_DO = SCH.NOTHING_TO_DO
def is_wrong_week(gp):
def score_statuses(fields, records, golds):

src/app.pyLocal UI

one invoice, one call, on 127.0.0.1:9003; renders with no key

# The minimal local UI. Standard library only -- python -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "9003"))
class H(BaseHTTPRequestHandler):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 6041 input and 15023 output tokens per query at top-k 1, of which about 150 tokens are fixed (system prompt, chat framing, the question itself) and the rest is retrieved context. Change the retrieval depth and the input scales with it. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model and top-k follow the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's invoices are entirely synthetic (tools/build_corpus.py, seed 20260823): no real carrier, shipper, lane, place, terminal, person, amount or currency exists in the corpus, the weekly fuel price series is an invention rather than a published index, and nothing was fetched from anywhere. No carrier tariff, freight contract or transport regulation is reproduced anywhere in the kit. The only outbound traffic the kit makes is one chat-completion request per invoice, carrying the mapped sections plus the fixed prompt and the shipped schedule -- and the two sections a real freight-audit desk would most mind sending anywhere, Carrier Contact (the carrier account, a named billing contact, a direct telephone extension and the shipper's own settlement account reference) and Prepared By, are mapped by no field and never leave the machine. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser.

The experimentWe did not attack it -- and the document is written by the party being audited

Three boundaries hold. Two were confirmed by reading the code: no code path adjusts an invoice, releases a payment or writes anything at all, and src/schedule.py reads four values off a row it is handed, so no invoice text can reach the decision. The third is confirmed by a CHECK RATHER THAN BY READING, and it had to be -- the fallback that would widen the wire to the carrier contact block never fires on this corpus, so evals/check_labels.py calls _fallback and for_field directly, on every document and every field plus a field name no hint matches, and asserts the two never-sent sections are absent from all of it. A guard that holds because nothing ever reaches it is not a guard. The fourth is open, and on this kit it is open in the worst direction: an indirect prompt injection needs a section an outside party authored, and here the outside party wrote the whole document. Confirmed by reading the code and by a pre-flight assertion that drives the fallback path directly, not by an attack run, on 2026-08-23 -- no attack run was fired; see redteam.why.

Three boundaries hold and the second is the one worth reading, because it holds by a check rather than by luck: the fallback path that would leak the carrier contact never fires on this corpus, so it is driven directly by the gold gate on every document before any run may spend. The fourth is open and is named as open -- ⚠︎ AND ON THIS KIT IT IS OPEN WIDER THAN ON MOST. The document under audit is written by the party being audited. A carrier that wanted a line cleared would not need to change a number; it would need one sentence in its own remarks field to make the reading come out matches, and nothing here has tried it.

The result0 attack trials. Three of four boundaries checked here hold on evidence -- and the privacy one holds because a gate drives a code path this corpus never reaches, not because the path was never taken. The one that does not hold is the one this kit's shape invites hardest: the invoice is written by the carrier, the finding is against the carrier, and whether a sentence in its own remarks field could make a line read matches is unmeasured.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260823), so no text in it came from an outside party and there was nothing adversarial to resist. The planted decoys measure whether the run reports the invoice's own shipments and no others, whether it separates a capped line and an all-in lane from a finding, and whether it declines a line it cannot price. None of those measures whether it obeys an instruction hidden in the same file -- and on this kit the cheap attack is not to change a call, it is to make one line read matches, on a document the counterparty authored.

HonestyWhat this does not prove

• Whether an indirect prompt injection in a Carrier Remarks line could move a call. No attack run exists, and the make-it-read-matches version is the one this kit's own shape suggests would be cheapest, because a cleared line is money nobody looks for again.
• Whether the kit behaves safely against a hostile provider -- a response body crafted to break the roster parsing in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed invoice, and nothing beyond that was tested.
• Whether a real invoice would carry anything sensitive this kit mishandles beyond the carrier contact block. The corpus has no personal data by construction; a real freight invoice carries negotiated rates, lane volumes, contract references and named contacts on both sides, and none of that path is exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_claim fires when the invoice carries at least one exception line -- `wrong_index_week` or `variance` -- whose variance is POSITIVE, meaning the carrier billed more than the schedule and the index support. An underbilled line is a finding and is deliberately NOT a claim: this kit never proposes paying a carrier more than it invoiced. Every value comes out of the same reply; the rule runs afterwards in pure code, over whatever the model returned, under the contract terms the model itself read, and never over gold. An invoice carrying a line the rule cannot read returns None rather than False -- an unknown is not a pass.

src/schedule.py::needs_claim(), called by extract() on every invoice, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by identical code. The six-gate rule it reads lives in the same module, which is also what the corpus generator wrote gold with and what src/prompt.py states to the model -- one definition, three readers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a claims policy. No carrier contract, shipper procedure or dispute process was consulted and none is reproduced. A real freight-audit desk weighs the size of the claim against the cost of raising it, the age of the invoice against the contract's claim window, and the state of the relationship with the carrier. None of that is here.
• It is NOT an action. carrier-payment-release is a non-configurable cap: no code path in this repository adjusts an invoice, approves or holds a payment, or files a claim. The condition is a value in a JSON response.
• It is NOT a check on the model. It is a predicate over statuses and variances that pure code already computed; if those are wrong it is confidently wrong with them.
• It is NOT the same thing as the pattern reading. systematic says one wrong basis was applied across the invoice; needs_claim says money is owed back. An invoice can be systematic and owe nothing if every wrong-week line was UNDERbilled.
• It is NOT a threshold anybody supplied. The pattern rule's 'two or more lines sharing a common week' is this kit's own choice, named as such in data/schedule.json, because it is the smallest condition mechanically checkable off one invoice. What counts as systematic for a claim your entity would actually raise has not been supplied.

WatchedWhat is watched, and why that one

The kit measures 27 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 20 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 50, extraction_cells 4284, failures 0, refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

2 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 2 runs measured the same system — they differ on provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 2 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

on every run of evals/run.py, and on every click of the local UI. Free on the floor.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the schedule and the index series are a JSON file read with json, money is held as integer minor units and index prices as integer thousandths rather than through a decimal library, the date arithmetic is datetime.timedelta, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

One path per invoice -- segment, select, prompt, one call, parse, span, the lag rule, the bracket lookup, the arithmetic, the gates -- with no branch, no loop and no agent. Two things differ from a flat-record extraction kit: the reply is a ROSTER rather than one object, so its length tracks the line count and the scorer has to align rows by key; and FIVE of the thirteen per-line fields are DERIVED rather than read, so the reasoning cost scales with the roster too. 93.3 pct of the output on this run was that reasoning.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain -- the JSON extraction from a fenced reply, the roster parsing, the backoff policy, the section regex, the money parser, the date arithmetic and the .env reader are all code somebody has to own.
• ⚠︎ AND THIS KIT PAID THAT BILL IN PUBLIC, TWICE, ON THE PART NOTHING VALIDATES. Its own field schema and its own rule disagree about when a recomputed_surcharge exists -- the schema says return one whenever the linehaul and the percentage are known, the rule returns at gate 2 when the BILLED amount is missing -- and 5 of the run's 9 non-hit cells are the model obeying the schema and being scored against the rule. Nothing anywhere compares the two documents. A structured-output schema shared between the prompt and the validator would have made the contradiction impossible to write.
• No framework means no framework's ecosystem -- no tracing, no eval harness beyond the one in evals/, no prompt registry, no structured-output enforcement. The reply is a ROSTER and nothing validates its shape except src/prompt.py::parse, which fails closed to ({}, []) rather than to an empty invoice, because 'nothing to recover on this invoice' is exactly the wrong thing to say by accident.
• The corpus generator, the gold gate and the scorer are three separate programs that must agree, and only convention holds them together -- one shared import of src/schedule.py. It works, and it is why every published label is the rule's own output rather than a typed value; it is also three places to change when the rule changes.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-fuel-surcharge on the fast tier, 2026-08-23. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 1 earlier run on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 1 run on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT (this repository's own licence). Nothing was fetched and nothing is licensed from anybody: no real carrier, shipper, index, place, terminal, person or currency is named anywhere in the kit, and the weekly fuel price series it recomputes against is an invention rather than a published series. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe index row, traced -- week and published price as ONE answer

Did the reply name the week the contract's lag rule selects AND the price published for it, together, for each line whose shipment date is stated? A right week citing the wrong price is a wrong citation, not half of one. THIS IS THE CLAIM THIS KIT EXISTS TO MAKE: a surcharge computed from the wrong week is arithmetically indistinguishable from a correct one, so the row is the only thing that separates them.

MethodHow the test was run

python -m evals.run --run-id <id> --yes

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

⚑ THIS IS THE KIT'S WHOLE THESIS HAPPENING LIVE, AND IT IS THE ONLY GENUINE DERIVATION ERROR IN 332 LINES. The model reads and derives; pure code decides. The model's own status would have put a correctly billed line on somebody's recovery worklist and sent a freight desk to argue with a carrier about 28.06 that was never owed. The published false-alarm rate on this run is 0.0 pct BECAUSE the rule ran over the model's values rather than accepting the model's verdict. It is also the row that shows why the citation is scored as a JOINT answer: the week and the price were both right and the reading off them was still wrong. ⚑ AND A FORKER WITH NO LABELS WOULD HAVE SEEN IT TOO. This is the ONE reply on the run whose own stated status disagreed with the schedule re-run over its own extracted values -- the no-gold consistency diagnostic, which is published as a Guardrails band and deliberately NOT as a grader, because it needs no reference answer to compute and therefore grades nothing. It is blind to the failure that matters most here, though: a reply that derives the wrong week and then reasons correctly from it agrees with itself perfectly.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, every label derived by running src/schedule.py over the values each invoice states under its own contract terms; evals/check_labels.py re-runs the rule over all 332 lines and drives the lag rule directly on each of them before any run may spend.

How tight can the band be? No tolerance band. A date string and a three-decimal price, both exact, both required.

Cadence: on every run of evals/run.py; free on --stub and --baseline.

The decisionWhen to reach for it

In one lineWrong-week recall -- the population an amount-based check cannot see

Of the 54 lines rated off a real published week that is not the one the contract selects, how many reached the recovery worklist at all, and how many were additionally given the RIGHT exception status -- which is what lets somebody show the carrier the row it used beside the row the contract selects.

MethodHow the test was run

python -m evals.run --run-id <id> --yes

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

⚑ THIS IS THE KIT'S WHOLE THESIS HAPPENING LIVE, AND IT IS THE ONLY GENUINE DERIVATION ERROR IN 332 LINES. The model reads and derives; pure code decides. The model's own status would have put a correctly billed line on somebody's recovery worklist and sent a freight desk to argue with a carrier about 28.06 that was never owed. The published false-alarm rate on this run is 0.0 pct BECAUSE the rule ran over the model's values rather than accepting the model's verdict. It is also the row that shows why the citation is scored as a JOINT answer: the week and the price were both right and the reading off them was still wrong. ⚑ AND A FORKER WITH NO LABELS WOULD HAVE SEEN IT TOO. This is the ONE reply on the run whose own stated status disagreed with the schedule re-run over its own extracted values -- the no-gold consistency diagnostic, which is published as a Guardrails band and deliberately NOT as a grader, because it needs no reference answer to compute and therefore grades nothing. It is blind to the failure that matters most here, though: a reply that derives the wrong week and then reasons correctly from it agrees with itself perfectly.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, every label derived by running src/schedule.py over the values each invoice states under its own contract terms; evals/check_labels.py re-runs the rule over all 332 lines and drives the lag rule directly on each of them before any run may spend.

How tight can the band be? A closed set of six statuses; nothing continuous, nothing to round.

Cadence: on every run of evals/run.py; free on --stub and --baseline.

The decisionWhen to reach for it

In one lineThe surcharge status, six-way

src/schedule.py run over the values the model returned, under the CONTRACT TERMS THE MODEL ITSELF READ, against gold's own status. It is not a tautology: a misread lag moves every line on an invoice at once, and a misread linehaul flips one -- so this measures the whole pipeline rather than the rule.

MethodHow the test was run

python -m evals.run --run-id <id> --yes

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

⚑ THIS IS THE KIT'S WHOLE THESIS HAPPENING LIVE, AND IT IS THE ONLY GENUINE DERIVATION ERROR IN 332 LINES. The model reads and derives; pure code decides. The model's own status would have put a correctly billed line on somebody's recovery worklist and sent a freight desk to argue with a carrier about 28.06 that was never owed. The published false-alarm rate on this run is 0.0 pct BECAUSE the rule ran over the model's values rather than accepting the model's verdict. It is also the row that shows why the citation is scored as a JOINT answer: the week and the price were both right and the reading off them was still wrong. ⚑ AND A FORKER WITH NO LABELS WOULD HAVE SEEN IT TOO. This is the ONE reply on the run whose own stated status disagreed with the schedule re-run over its own extracted values -- the no-gold consistency diagnostic, which is published as a Guardrails band and deliberately NOT as a grader, because it needs no reference answer to compute and therefore grades nothing. It is blind to the failure that matters most here, though: a reply that derives the wrong week and then reasons correctly from it agrees with itself perfectly.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, every label derived by running src/schedule.py over the values each invoice states under its own contract terms; evals/check_labels.py re-runs the rule over all 332 lines and drives the lag rule directly on each of them before any run may spend.

How tight can the band be? A closed set of six values. Exact.

Cadence: on every run of evals/run.py; free on --stub and --baseline.

The decisionWhen to reach for it

In one lineFalse-alarm rate on the recovery worklist

Of every line this kit put on the worklist, what share had a gold status of matches, capped_at_maximum or fuel_exempt_all_in -- the three answers that mean nothing needed doing. A not_determinable line that was flagged is counted separately as MISROUTED, because somebody does have to open that record; they have to open it for a different reason.

MethodHow the test was run

python -m evals.run --run-id <id> --yes

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

⚑ THIS IS THE KIT'S WHOLE THESIS HAPPENING LIVE, AND IT IS THE ONLY GENUINE DERIVATION ERROR IN 332 LINES. The model reads and derives; pure code decides. The model's own status would have put a correctly billed line on somebody's recovery worklist and sent a freight desk to argue with a carrier about 28.06 that was never owed. The published false-alarm rate on this run is 0.0 pct BECAUSE the rule ran over the model's values rather than accepting the model's verdict. It is also the row that shows why the citation is scored as a JOINT answer: the week and the price were both right and the reading off them was still wrong. ⚑ AND A FORKER WITH NO LABELS WOULD HAVE SEEN IT TOO. This is the ONE reply on the run whose own stated status disagreed with the schedule re-run over its own extracted values -- the no-gold consistency diagnostic, which is published as a Guardrails band and deliberately NOT as a grader, because it needs no reference answer to compute and therefore grades nothing. It is blind to the failure that matters most here, though: a reply that derives the wrong week and then reasons correctly from it agrees with itself perfectly.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, every label derived by running src/schedule.py over the values each invoice states under its own contract terms; evals/check_labels.py re-runs the rule over all 332 lines and drives the lag rule directly on each of them before any run may spend.

How tight can the band be? A binary over a closed status set. No band.

Cadence: on every run of evals/run.py; free on --stub and --baseline.

The decisionWhen to reach for it

In one linePer-(line, field) exact match, light normalisation

Does the model's value for each of the 12 scored per-line fields match gold after trimming whitespace and punctuation, plus six invoice-level cells per record? THE DENOMINATOR IS GOLD'S LINES, NOT THE RUN'S: a shipment the run never returned would score as a miss on all twelve of its cells rather than disappearing.

MethodHow the test was run

python -m evals.run --run-id <id> --yes

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

⚑ THIS IS THE KIT'S WHOLE THESIS HAPPENING LIVE, AND IT IS THE ONLY GENUINE DERIVATION ERROR IN 332 LINES. The model reads and derives; pure code decides. The model's own status would have put a correctly billed line on somebody's recovery worklist and sent a freight desk to argue with a carrier about 28.06 that was never owed. The published false-alarm rate on this run is 0.0 pct BECAUSE the rule ran over the model's values rather than accepting the model's verdict. It is also the row that shows why the citation is scored as a JOINT answer: the week and the price were both right and the reading off them was still wrong. ⚑ AND A FORKER WITH NO LABELS WOULD HAVE SEEN IT TOO. This is the ONE reply on the run whose own stated status disagreed with the schedule re-run over its own extracted values -- the no-gold consistency diagnostic, which is published as a Guardrails band and deliberately NOT as a grader, because it needs no reference answer to compute and therefore grades nothing. It is blind to the failure that matters most here, though: a reply that derives the wrong week and then reasons correctly from it agrees with itself perfectly.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, every label derived by running src/schedule.py over the values each invoice states under its own contract terms; evals/check_labels.py re-runs the rule over all 332 lines and drives the lag rule directly on each of them before any run may spend.

How tight can the band be? Exact match after trimming whitespace and punctuation. Integer comparison for the variance; a closed set of six values for the status. Nothing continuous.

Cadence: on every run of evals/run.py; free on --stub and --baseline.

The decisionWhen to reach for it

In one lineThe recomputation arithmetic (the model's own)

The MODEL's own recomputed_surcharge against the derivation -- the linehaul times the applicable percentage, rounded half up to the minor unit. No invoice states it. The kit publishes the CODE's figure and this number decides nothing, which is exactly what makes it a measurement of arithmetic rather than of copying.

MethodHow the test was run

python -m evals.run --run-id <id> --yes

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

⚑ THIS IS THE KIT'S WHOLE THESIS HAPPENING LIVE, AND IT IS THE ONLY GENUINE DERIVATION ERROR IN 332 LINES. The model reads and derives; pure code decides. The model's own status would have put a correctly billed line on somebody's recovery worklist and sent a freight desk to argue with a carrier about 28.06 that was never owed. The published false-alarm rate on this run is 0.0 pct BECAUSE the rule ran over the model's values rather than accepting the model's verdict. It is also the row that shows why the citation is scored as a JOINT answer: the week and the price were both right and the reading off them was still wrong. ⚑ AND A FORKER WITH NO LABELS WOULD HAVE SEEN IT TOO. This is the ONE reply on the run whose own stated status disagreed with the schedule re-run over its own extracted values -- the no-gold consistency diagnostic, which is published as a Guardrails band and deliberately NOT as a grader, because it needs no reference answer to compute and therefore grades nothing. It is blind to the failure that matters most here, though: a reply that derives the wrong week and then reasons correctly from it agrees with itself perfectly.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, every label derived by running src/schedule.py over the values each invoice states under its own contract terms; evals/check_labels.py re-runs the rule over all 332 lines and drives the lag rule directly on each of them before any run may spend.

How tight can the band be? Exact to the minor unit. The kit's own tolerance of 1 minor unit applies to the STATUS gate, not to this comparison.

Cadence: on every run of evals/run.py; free on --stub and --baseline.

The decisionWhen to reach for it