The business caseThe problem this solves

A dealer has transmitted a warranty claim and someone has to decide whether it is payable and at what labour time before the money moves. That decision is not a date subtraction -- it is a DENIAL YOU WILL HAVE TO DEFEND WHEN THE DEALER APPEALS IT, and one you may have to reverse in front of a bulletin the manufacturer issued itself. So the failure that costs is not a wrong coverage window. It is a line denied on a window the pack has already reopened: a special coverage adjustment that extended that operation on that model, a technical hotline case that authorised the extra diagnostic time, a prior repair carried out under the district manager's own goodwill authorisation and held at the region rather than stapled to the repair order. The coverage really is closed in all three and the denial is still wrong. And the other direction costs the manufacturer instead: a combination operation, where the hours claimed equal the published time exactly and the published time is not the time payable, reads as perfect line by line. Over-payment is fraud exposure; wrong denial is a dealer relations failure, and on this corpus every arm makes one of them. A warranty analyst reading a transmitted claim line by line against the policy: working out completed months from the in-service date to the repair date and comparing the odometer against the limit for whichever coverage governs that operation, looking each operation code up in the labour time guide and comparing the hours claimed against the published time, classifying each cause code as a covered defect or as wear, maintenance or customer damage, checking the file for a prior repair to the same component that nobody authorised, and then -- the part no table helps with -- reading the service correspondence for the one sentence that reverses any of it.

The inputThe actual warranty claim review packs

The corpus is 45 warranty claim review packs, 0.43 MB (txt 45). A dealer warranty claim carries a VIN, a named customer, a dealer code, a repair order and a labour rate that is commercially confidential between a manufacturer and its network, and the policy and labour time guide it is adjudicated against are the manufacturer's own. There is no public set of (claim, adjudication) pairs and there is not going to be one, and publishing a scrubbed real set would be worse than publishing none, because the scrubbing is precisely where the interesting defect hides -- the sentence naming a special coverage adjustment is the first thing a scrubber removes and the only thing that makes the case a case. The harder reason is that the thing being measured has to be PLANTED to be measured at all. The question this kit asks is not "is this claim outside its coverage" but "is this a denial you could defend when the dealer appeals it holding the manufacturer's own document", and a real archive does not come labelled with which apparent denial grounds had already been reopened by a bulletin, a hotline case or a district manager. So the corpus is generated, the generator ships, the seed is recorded, and the answer key is re-derived by a gate rather than trusted.

Warranty Claim Review
----------------------------------------------------------------
  Claim               WC-0001
  Dealer              Harrowgate Motor Group, dealer code D-6613
  Vehicle             2024 Calder Ridgeline Hybrid
  VIN                 5JVXEB6FSXW7PVWD0
  Repair order        RO-252396
  Repair date         2026-02-28
  In service date     2023-12-28
  Odometer at repair  23,099
  Labour rate         134.50
  Claim basis         published labour operation times, coverage under the vehicle warranty policy

Warranty Policy And Guides
----------------------------------------------------------------

  Coverage terms

  CV-BAS    Bumper to bumper, basic vehicle               36 months      36,000 miles   WP-2.1
  CV-PWR    Powertrain, engine transmission and drive     60 months      60,000 miles   WP-2.2
  CV-EMI    Emissions related components                  96 months      80,000 miles   WP-2.3

  Labour operation guide

  LO-1120   Alternator, remove and replace              CV-BAS       1.60 hours   WP-3.1
  LO-1210   Door mirror assembly, replace               CV-BAS       0.70 hours   WP-3.1
  LO-1408   Air conditioning compressor, replace        CV-BAS       3.40 hours   WP-3.1
  LO-1470   Seat frame, front, remove and replace       CV-BAS       2.80 hours   WP-3.1
  LO-1512   Infotainment head unit, replace             CV-BAS       1.40 hours   WP-3.1
  LO-1580   Tailgate strut pair, replace                CV-BAS       0.60 hours   WP-3.1
  LO-2210   Cylinder head gasket, replace               CV-PWR      11.60 hours   WP-3.1
  LO-2255   Rocker cover gasket, replace                CV-PWR       1.10 hours   WP-3.1
  LO-2404   Transmission valve body, replace            CV-PWR       5.10 hours   WP-3.1

The outcomeWhat a good result looks like

A drafted adjudication: one entry per claimed line, in the pack's own order, each carrying DENY with one of six named grounds, the policy provision from this pack and the record identifiers behind it -- or PAYABLE, or INSUFFICIENT_EVIDENCE naming what is missing -- plus the labour time the operation guide publishes, read independently on EVERY line whether or not anything is wrong with it, and a claim-level recommendation. Beside every line, what pure code alone would have written, which on this corpus scores HIGHER than the paid arm.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Every fact that decides a claim on your bench is already a FIELD -- an in-service date, an odometer reading, a cause code in a table with a covered/not-covered column, an operation code in a published time guide, a prior-repair record with an authorisation number on it -- and your dealers claim labour in hundredths of an hour. — the free floor -- claim-gate, $0.00, and set TOL to your own rounding
  It scores 76.30 pct on the discriminator against the fast tier's 44.44, cites the provision correctly on 100 pct of its denials against 66.15, names all 16 unsettleable lines against 6, reads the published labour time right on 261 of 261 against 239, and invents nothing. It runs all 45 packs in 0.1 s with no key and no network.
• Your bulletins, hotline authorisations and goodwill decisions live in prose -- a technical information system, a CRM case, a region's spreadsheet -- and a claim that looks out of coverage on the printed policy is routinely reopened by one of them. — the paid arm, and budget for the over-denials
  This is the only column free code does not reach at any tolerance: 22 of 22 for the fast tier against 0 of 22 for all three floors, and the ablation proves it is real reading -- blind the same arm and it drops to 0 of 22 and lands on the floor's figure in five columns. It also cuts the expensive failure by three and a half times: 12 silent over-denials of 42 against the floor's 42 of 42.
• You are pointing this at real transmitted claim records out of a dealer management system. — rewrite src/claim.py first, and expect the correspondence column to disappear
  src/claim.py parses this corpus's fixed-column layout and returns EMPTY tables against anything else rather than guessing, so every line would come back INSUFFICIENT_EVIDENCE -- an honest failure, not a hidden one. And the half this kit measures as the paid arm's premium is exactly the half that is not in the claim feed at all in a real estate. data/SOURCES.md says what breaks, field by field.

And where nothing here is good enough:

• You want one number to decide with. — neither -- read the column split
  ⚠︎ THE SINGLE NUMBER IS THE MOST MISLEADING THING ON THIS PAGE. The assigned floor ties every arm on the date column (20 of 20) and the mileage column (18 of 18) and scores 0 of 34 and 0 of 16 on the two cause columns. The strongest floor closes both cause columns for nothing and still takes 0 of 22 on the combination-operation half of labour time. The paid arm takes that 22 and pays for it with 62 over-denials. No single accuracy figure carries any of that.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-26. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-26 — r001-warranty-claim. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Clone, python3 -m src.app, open 127.0.0.1:9074. No key, no pip install, no index build, no network. The corpus, the answer key, all three free floors, the tolerance sweep, the answer-key gate and every recorded result are committed, so the whole comparison re-scores offline for nothing -- and the arm that scores highest on this corpus needs no key at all: b002-warranty-claim-claimgate reads 76.30 per cent on the discriminator against r001-warranty-claim's 44.44, over the same 270 lines.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. Model call only, one per claim, 5 concurrent workers against a key sibling kits were using at the same time; 1,503.4 s of wall clock for 45 claims, 45 of 45 answered, 0 truncated. Read it as a bound rather than a clean single-tenant latency. The correspondence-blind arm ran the same 45 at 6 workers in 1,072.7 s for a p50 of 142,311 ms and a p95 of 212,947 ms, so the spread across concurrency settings is visible rather than asserted.

Current processWhat it replaces

A warranty analyst reading a transmitted claim line by line against the policy: working out completed months from the in-service date to the repair date and comparing the odometer against the limit for whichever coverage governs that operation, looking each operation code up in the labour time guide and comparing the hours claimed against the published time, classifying each cause code as a covered defect or as wear, maintenance or customer damage, checking the file for a prior repair to the same component that nobody authorised, and then -- the part no table helps with -- reading the service correspondence for the one sentence that reverses any of it.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusEvalsEvalsArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• 4 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one claimed line is

45 packs, each one transmitted claim against one repair order, carrying the warranty policy and guides, six claimed lines, the records on file and the service correspondence. No claim references another and nothing carries between them, so the 45 calls can be fired in any order or all at once, and a pack that fails tells you nothing about the 44 beside it. The composition is CONSTRUCTED rather than dealt: 130 of the 270 lines are DENY, 124 PAYABLE and 16 INSUFFICIENT_EVIDENCE, and at claim level 37 packs are RETURN_FOR_ADJUSTMENT against 4 REQUEST_SUPPORT and 4 RECOMMEND_PAY_AS_CLAIMED. That skew is stated rather than hidden, because it is exactly why the single majority word already scores 82.22 per cent on the claim action and why both the majority and the measured figure are printed side by side. The line mix carries 82 clean lines, 108 lines whose denial ground the tables settle, 22 whose denial ground only a sentence settles, 42 reverse traps the correspondence has already answered and 16 lines that settle either way.

SetupWhat the setup figure measured

THERE IS NO INDEX AND NO RETRIEVAL IN THIS KIT, and that is a design decision rather than an omission. One claim pack is read whole and sent whole, minus the withheld Dealer Position block -- no chunking, no embedding, no nearest-neighbour lookup, no pre-digest. The reason is the thing being measured: the answer to a line is often a single sentence in the Service Correspondence which contradicts a printed table, and any step that summarises, ranks or truncates the pack is a place for that sentence to be lost before the model ever sees it, after which the measurement is of the retriever. Nothing is built, so nothing is spent building it. For scale: the strongest free floor scores all 45 packs in 0.1 seconds of wall clock, and the whole comparison re-runs offline for nothing.

LicenceLicence

MIT -- this repository's own licence. The corpus has no third-party component to licence separately: it is written by a generator that ships in the repo, from a seed recorded in data/corpus-stats.json, and every figure in it is invented. Verified by reading every generator input on 2026-08-26.

Bring your ownBring your own warranty claim review packs

tools/build_corpus.py writes data/corpus/*.txt, data/gold.jsonl and data/corpus-stats.json. To point the kit at your own claims, work in this order: src/segment.py's heading pattern if your sections are not underlined; src/claim.py's regular expressions for the coverage terms, the operation guide, the cause classification and the line and record blocks; src/select.py's NEVER_SENT for whichever block of yours is the Dealer Position; src/checks.py's six structured checks, its SUPPORT_TYPE mapping AND its TOL -- run python3 -m evals.tolerance against your own claims before trusting the value shipped here, which was chosen by that sweep on THIS corpus. Then write your own data/gold.jsonl and run python3 -m evals.check_labels over it, because every number this kit publishes is a comparison against that file and it is the piece that cannot be skipped. evals/scoring.py, evals/run.py and the UI are shape-independent and carry over unchanged.

⚠︎ And what stops being true when you do: src/select.py withholds a NAMED SECTION and is not a redaction system, and this corpus contains its own counter-example. The Dealer Position block on WC-0042 states that the district manager may pay the claim as goodwill up to 610.50 without a region review, and that block never leaves the machine. Six lines earlier, the Service Correspondence records that the prior repair on line L3 was carried out under district manager goodwill authorisation GW-28842 -- and that sentence IS sent, because it is the only thing in the pack that makes the line payable. The kit cannot both withhold the commercial posture and read the correspondence, and it chooses to read the correspondence. If you point this at real claims, that is the first seam to look at, and the fix is a redaction pass rather than a longer denylist.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 2,970 tokens assembled, 1,846 are evidences — 62% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The whole thing, byte for byte, as src/prompt.py assembled it for WC-0042 and src/adapters posted it: the system string followed by the claim pack. WC-0042 is one of the two claims p001-warranty-claim-prompt-tokens actually measured, so the token split below is against THIS prompt rather than against an average of prompts nobody sent. Nothing is elided and nothing is summarised -- a pre-digest is a place for the operative sentence to be lost before the model sees it, after which the measurement is of the summariser.

Full promptThe prompt in full

You are adjudicating a dealer warranty claim for a vehicle manufacturer.

You act for the MANUFACTURER, before a decision. An adjudication is not a payment and not a denial:
it is what the warranty analyst reads before deciding what to do with the claim. A denial you
cannot support comes back as a dealer appeal with the document that reverses it, and one of those
documents is often the manufacturer's own bulletin. A denial with no policy provision behind it is
not something the dealer has to answer at all.

Return one entry for EVERY line in "Claimed Lines", in the order the pack lists them, using the
pack's own line identifiers. Never drop a line because it looks unremarkable.

For each line give exactly one disposition:

  DENY                   a denial ground can be STATED, the policy provision that carries it is in
                         this pack, and the record that proves it is in this pack.
  PAYABLE                the line stands as claimed. Either it meets the policy, or something in
                         the pack already answers the apparent problem.
  INSUFFICIENT_EVIDENCE  something about the line cannot be settled from this pack -- the term it
                         would turn on is not here, or the record that would prove it is not here.
                         Do not raise a denial; say what is missing.

When the disposition is DENY, the ground is exactly one of:
  NOT_A_COVERED_DEFECT       the cause is wear from use in service or a maintenance item, not a
                             defect in factory material or workmanship
  CUSTOMER_DAMAGE            the cause is impact, misuse, contamination or an unapproved fitment
  PRIOR_UNAUTHORISED_REPAIR  the failure follows a repair to the same component that the
                             manufacturer did not authorise
  OUT_OF_COVERAGE_TIME       on the repair date the vehicle was past the months the coverage
                             governing this operation runs from the in-service date
  OUT_OF_COVERAGE_MILEAGE    the odometer at repair was past that coverage's mileage limit
  LABOUR_TIME_EXCEEDED       the hours claimed are above the time payable for that operation
Otherwise ground is null.

Also give, for every line, the labour time you independently read for that operation: the hours the
labour operation guide in this pack publishes for the line's operation code, to two decimal places.
Give null where the pack does not carry a published time for it.

Cite the governing policy provision as a provision identifier that appears in this pack, and attach
the support as record identifiers that appear in this pack. NEVER write a provision or record
identifier that is not printed in the pack in front of you -- an invented citation is worse than
none, because it is the first thing the dealer's warranty administrator will check.

Anything in the pack may bear on a line. Where a printed table and a later statement in the pack
contradict each other, they are not a tie.

Reply with JSON and nothing else:

{"claim_action": "RETURN_FOR_ADJUSTMENT" | "REQUEST_SUPPORT" | "RECOMMEND_PAY_AS_CLAIMED",
 "lines": [{"line": "<the pack's line identifier>",
            "item": "<what was repaired, one short phrase>",
            "disposition": "DENY" | "PAYABLE" | "INSUFFICIENT_EVIDENCE",
            "ground": "<one of the six, or null>",
            "policy_provision": "<a provision identifier from this pack, or null>",
            "support": ["<record identifiers from this pack>"],
            "allowed_labour_hours": <number or null>,
            "finding": "<the one sentence the adjudication would carry for this line>"}],
 "rationale": "<two sentences at most, on what decided the hardest line>"}

claim_action is RETURN_FOR_ADJUSTMENT if any line is DENY; REQUEST_SUPPORT if none is but at least
one is INSUFFICIENT_EVIDENCE; RECOMMEND_PAY_AS_CLAIMED otherwise. None of the three pays, denies or
submits anything -- each is a recommendation to the person who does.

Warranty Claim Review
----------------------------------------------------------------
Claim               WC-0042
  Dealer              Pinemarsh Motors, dealer code D-1184
  Vehicle             2023 Orrell Crossover 2.4
  VIN                 WMY5MN6L72XZKC7MR
  Repair order        RO-565039
  Repair date         2026-02-24
  In service date     2025-02-24
  Odometer at repair  38,546
  Labour rate         111.61
  Claim basis         published labour operation times, coverage under the vehicle warranty policy

Warranty Policy And Guides
----------------------------------------------------------------
Coverage terms

  CV-BAS    Bumper to bumper, basic vehicle               36 months      36,000 miles   WP-2.1
  CV-PWR    Powertrain, engine transmission and drive     60 months      60,000 miles   WP-2.2
  CV-EMI    Emissions related components                  96 months      80,000 miles   WP-2.3

  Labour operation guide

  LO-1120   Alternator, remove and replace              CV-BAS       1.60 hours   WP-3.1
  LO-1145   Starter motor, remove and replace           CV-BAS       1.30 hours   WP-3.1
  LO-1265   Window regulator, front door, replace       CV-BAS       1.90 hours   WP-3.1
  LO-1310   Instrument cluster, replace and configure   CV-BAS       1.10 hours   WP-3.1
  LO-1512   Infotainment head unit, replace             CV-BAS       1.40 hours   WP-3.1
  LO-1580   Tailgate strut pair, replace                CV-BAS       0.60 hours   WP-3.1
  LO-2255   Rocker cover gasket, replace                CV-PWR       1.10 hours   WP-3.1
  LO-2372   Water pump, replace                         CV-PWR       3.20 hours   WP-3.1
  LO-2404   Transmission valve body, replace            CV-PWR       5.10 hours   WP-3.1
  LO-2519   Drive shaft, front, replace                 CV-PWR       2.30 hours   WP-3.1
  LO-2566   Engine mount, replace                       CV-PWR       1.50 hours   WP-3.1
  LO-3140   Catalytic converter, replace                CV-EMI       2.60 hours   WP-3.1
  LO-3277   Exhaust gas recirculation valve, replace    CV-EMI       1.00 hours   WP-3.1

  Cause classification

  CZ-DEF    Defect in factory material or workmanship                   covered defect    WP-4.1
  CZ-ASM    Assembly fault found at the factory build stage             covered defect    WP-4.1
  CZ-SFT    Control software fault corrected by the published update    covered defect    WP-4.1
  CZ-WER    Component worn from normal use in service                   wear              WP-4.2
  CZ-MNT    Maintenance item replaced at its published interval         maintenance       WP-4.2
  CZ-CDM    Impact, misuse, contamination or an unapproved fitment      customer damage   WP-4.3

  Provision index

  WP-2.1    Basic vehicle coverage runs the months and miles in the coverage terms table
  WP-2.2    Powertrain coverage runs the months and miles in the coverage terms table
  WP-2.3    Emissions coverage runs the months and miles in the coverage terms table
  WP-3.1    Labour is payable at the time the labour operation guide publishes
  WP-3.4    Time above the published operation is payable only with a hotline authorisation
  WP-4.1    A defect in factory material or workmanship is a covered repair
  WP-4.2    Wear from use in service and maintenance items are not covered repairs
  WP-4.3    Damage from impact, misuse, contamination or an unapproved fitment
  WP-5.2    A failure following a repair the manufacturer did not authorise is not payable
  WP-6.1    Coverage runs from the date the vehicle first entered service

Claimed Lines
----------------------------------------------------------------
Line L1
  Description         Transmission valve body, replace (LO-2404)
  Cause code          CZ-ASM
  Labour claimed      5.13
  Labour rate         111.61
  Labour amount       572.56
  Parts amount        1,605.95
  Total claimed       2,178.51

Line L2
  Description         Rocker cover gasket, replace (LO-2255)
  Cause code          CZ-DEF
  Labour claimed      1.10
  Labour rate         111.61
  Labour amount       122.77
  Parts amount        1,480.28
  Total claimed       1,603.05

Line L3
  Description         Drive shaft, front, replace (LO-2519)
  Cause code          CZ-ASM
  Labour claimed      2.32
  Labour rate         111.61
  Labour amount       258.94
  Parts amount        763.04
  Total claimed       1,021.98

Line L4
  Description         Exhaust gas recirculation valve, replace (LO-3277)
  Cause code          CZ-DEF
  Labour claimed      1.00
  Labour rate         111.61
  Labour amount       111.61
  Parts amount        165.38
  Total claimed       276.99

Line L5
  Description         Fuel tank assembly, replace (LO-4160)
  Cause code          CZ-ASM
  Labour claimed      2.60
  Labour rate         111.61
  Labour amount       290.19
  Parts amount        514.66
  Total claimed       804.85

Line L6
  Description         Water pump, replace (LO-2372)
  Cause code          CZ-ASM
  Labour claimed      3.20
  Labour rate         111.61
  Labour amount       357.15
  Parts amount        1,008.76
  Total claimed       1,365.91

Records On File
----------------------------------------------------------------
Record RC-0042-01
  Type                Delivery record
  Reference           WC-0042
  Date                2025-02-24
  Content             Delivery and registration record for this vehicle
  In service date     2025-02-24
  Signed by           Selling dealer

Record RC-0042-02
  Type                Odometer record
  Reference           WC-0042
  Date                2026-02-24
  Content             Odometer reading recorded at check-in for this repair order
  Recorded odometer   38,546
  Signed by           Dealer service advisor

Record RC-0042-03
  Type                Technician story
  Reference           L1
  Date                2026-02-24
  Content             Complaint, cause and correction as recorded on the repair order
  Cause narrative     Transmission valve body inoperative from cold; the assembly was found out of specification as built; transmission valve body replaced and the system retested.
  Signed by           Dealer service technician

Record RC-0042-04
  Type                Technician story
  Reference           L3
  Date                2026-02-24
  Content             Complaint, cause and correction as recorded on the repair order
  Cause narrative     Drive shaft found faulty on arrival; the fault traces to how the unit was put together at build rather than to anything done in service; drive shaft replaced.
  Signed by           Dealer service technician

Record RC-0042-05
  Type                Prior repair
  Reference           L3
  Date                2025-09-19
  Content             An earlier repair to the same component on this vehicle
  Repair order        RO-307259
  Component           Drive shaft, front, replace
  Performed by        Independent repairer
  Authorisation       none

Record RC-0042-06
  Type                Technician story
  Reference           L5
  Date                2026-02-24
  Content             Complaint, cause and correction as recorded on the repair order
  Cause narrative     Fuel tank assembly found faulty on arrival; the fault traces to how the unit was put together at build rather than to anything done in service; fuel tank assembly replaced.
  Signed by           Dealer service technician

Service Correspondence
----------------------------------------------------------------
- The dealer's warranty administrator has held the manufacturer's certification since 2011.
- The customer declined the multi-point inspection recommendations at this visit.
- Recall 21V-114 was checked against this VIN and is not open.
- Parts return tag PR-260752 was raised and the failed part shipped to the regional return centre.
- The prior repair recorded against L3 was carried out under district manager goodwill authorisation GW-28842; the authorisation is held at the region and was not attached to the repair order.
- The repair order was reopened once to correct a typing error in the customer address.

Raw responseThe raw response

{"claim_action":"RETURN_FOR_ADJUSTMENT","lines":[{"line":"L1","item":"Transmission valve body, replace","disposition":"DENY","ground":"LABOUR_TIME_EXCEEDED","policy_provision":"WP-3.4","support":["RC-0001-03"],"allowed_labour_hours":5.10,"finding":"The claimed 5.11 hours on LO-2404 exceeds the published 5.10 hours and no hotline authorisation appears, so the excess labour is not payable."},{"line":"L2","item":"Oxygen sensor, upstream, replace","disposition":"PAYABLE","ground":null,"policy_provision":"WP-4.1","support":["RC-0001-04"],"allowed_labour_hours":0.80,"finding":"The upstream oxygen sensor out-of-spec as built is a covered factory assembly defect within CV-EMI and the labour matches the published 0.80 hours."},{"line":"L3","item":"Clutch assembly, replace","disposition":"PAYABLE","ground":null,"policy_provision":"WP-4.1","support":["RC-0001-05"],"allowed_labour_hours":7.80,"finding":"The clutch assembly internal failure is a covered defect within CV-PWR and the claimed 7.80 hours matches the published time."},{"line":"L4","item":"Cylinder head gasket, replace","disposition":"DENY","ground":"PRIOR_UNAUTHORISED_REPAIR","policy_provision":"WP-5.2","support":["RC-0001-07"],"allowed_labour_hours":11.60,"finding":"The same cylinder head gasket was previously repaired by an independent repairer without authorisation, so WP-5.2 excludes this failure."},{"line":"L5","item":"Engine mount, replace","disposition":"DENY","ground":"LABOUR_TIME_EXCEEDED","policy_provision":"WP-3.4","support":["RC-0001-09"],"allowed_labour_hours":0.70,"finding":"Because LO-2455 and LO-2566 are claimed together, LO-2566 is limited to the 0.70-hour combination time, and the claimed 1.50 hours exceeds it."},{"line":"L6","item":"Rocker cover gasket, replace","disposition":"DENY","ground":"PRIOR_UNAUTHORISED_REPAIR","policy_provision":"WP-5.2","support":["RC-0001-11"],"allowed_labour_hours":1.10,"finding":"The same rocker cover gasket was previously repaired by an independent repairer without authorisation, so WP-5.2 excludes this failure."}],"rationale":"L5 is the hardest line: the later combination notice supersedes the LO-2566 table time, so only 0.70 hours is allowed for the engine mount. L4 and L6 are separately excluded because the same components were previously repaired by an un-authorised independent repairer."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Decide a car dealer's warranty claim, line by line — 270 warranty claim review packs drawn from 45 real warranty claim review packs. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Pure code, exact match per claimed line against data/gold.jsonl. NO MODEL GRADES ANYTHING on this kit -- there is no judge, no rubric and no second call. evals/scoring.py compares the arm's answer to the key and every figure on this page falls out of that comparison.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on this page is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid. A PROJECTION CARD, not a bill. The provider that ran this publishes no rate card this repo commits, so every dollar figure on this page is measured tokens times a published rate for a different model. Swap the card and every figure moves.

Rates checked 2026-08-26. The provider that actually ran every call on this kit is kept off this page per the series rule; the real spend sits in the shared call ledger. And the arm that scores HIGHEST on this corpus is not priced at all because it costs nothing: the three free floors, the tolerance sweep, the answer-key gate and the wiring stub make no call.

Grading unitWhat the grading figure prices

EVERY GRADER ON THIS KIT IS FREE. Scoring is pure code -- exact match per claimed line against data/gold.jsonl -- so re-scoring all four arms costs $0.00, sends nothing anywhere and needs no key. The unit the kit is PRICED in is one transmitted claim covering every line claimed on it, and that is in the Cost lens; grading it is free.

The gradersThree ways to grade

⚑ THE ASSIGNED FLOOR IS b000-warranty-claim-coveragewindow AND IT IS THE WEAKEST OF THE THREE, WHICH IS THE POINT OF SHIPPING ALL THREE. A deterministic in-coverage check by date and mileage plus an operation-code lookup takes 34.81 pct. It cannot state which of six grounds it is looking at -- a wear item, an unapproved fitment and a prior repair nobody authorised all sail through it as payable -- and it cites nothing, so 0 of its denials carry a provision or a record. Reporting only that number would have flattered the model by 9.63 points. The honest floor is the strongest thing free code can do, which is claim-gate at 76.30 pct. ⚑ THE FLOOR THIS KIT WAS TOLD TO BEAT IS b000-warranty-claim-coveragewindow: a deterministic in-coverage check by date and mileage, plus the operation-code lookup. Split the 130 unpayable lines by what actually decides them and the two arms do not compete -- they are good at different columns, and averaging them into one accuracy hides the whole finding. Out of coverage by TIME -- completed months against the term -- assigned floor 100.0 pct, strongest floor 100.0 pct, fast tier 100.0 pct, of 20. Date arithmetic, and everything ties. Nobody should pay for this column. Out of coverage by MILEAGE -- odometer against the limit -- assigned floor 100.0 pct, strongest floor 100.0 pct, fast tier 100.0 pct, of 18. A single integer comparison, and everything ties. Same conclusion. Labour time above the published operation -- assigned floor 47.62 pct, strongest floor 47.62 pct, fast tier 100.0 pct, of 42. The op-code lookup takes the 20 lines where the hours claimed are visibly above the published time and NONE of the 22 where they equal it exactly and a combination note in the correspondence supersedes it. Both floors stop at the same 47.62 pct because the second half is not in any table. Cause not a covered defect -- wear, maintenance or customer damage -- assigned floor 0.0 pct, strongest floor 100.0 pct, fast tier 100.0 pct, of 34. ⚠︎ THE ASSIGNED FLOOR SCORES ZERO. A coverage window has no opinion about why a part failed, so every wear item and every unapproved fitment sails through it as payable. Reading the cause classification table closes it completely -- which is what the strongest floor does, for nothing. Failure after a prior repair nobody authorised -- assigned floor 0.0 pct, strongest floor 100.0 pct, fast tier 87.5 pct, of 16. ⚠︎ ZERO AGAIN for the assigned floor, and the one column where free code beats the paid arm outright on finding: the strongest floor takes 16 of 16 and the fast tier 14 of 16. So the split IS the finding, and it is not the one either extreme predicts. The assigned floor ties everything on the two pure date-and-mileage columns -- 20 of 20 and 18 of 18 -- and scores 0 of 34 and 0 of 16 on the two cause columns, which is the whole of its 34.81 pct. Adding the cause table and the prior-repair record is pure code and takes it to 76.30 pct. What is left after that is 22 lines in one column, the combination-operation half of labour time, where every printed figure is internally consistent and no free floor moves off 0 -- and the paid arm takes 22 of 22. Buy a model for that column or for none of them.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-26. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

100 live calls in total: 45 scored, 45 correspondence-blind, 4 on the discarded 32,000-token probe, 4 on the re-taken 64,000 probe, and 2 at max_tokens=1 for the prompt-token split. The three free floors, the tolerance sweep, the stub and the answer-key gate cost $0.00 and can be re-run by anyone with no key. ⚠︎ THE DISCARDED PROBE'S COST IS LISTED RATHER THAN NETTED OUT: it was paid for, its scores are not published, and hiding it would understate what this kit cost to build.

Cost driversWhat actually moves the bill

• THE OUTPUT SIDE, AND IT IS NOT CLOSE. 18,959.91 output tokens per claim against 3,316.00 input -- at this card the output is 96.9 pct of the bill.
• REASONING THE PROVIDER RE-ROLLS PER CALL. 818,949 of 853,196 output tokens across the scored run -- 96.00 pct -- are provider-side reasoning rather than the adjudication that gets published. The answer itself is about 760 tokens.
• The claim pack, which is two thirds of the input bill: 1,846 of WC-0042's 2,970 input tokens. The instruction is a fixed 1,124 on every call, measured as an intercept rather than counted.
• The number of claimed lines on a repair order, indirectly. Every pack here carries exactly six, so this kit CANNOT separate cost-per-claim from cost-per-line and does not pretend to.
• Nothing else. There is no index, no retrieval, no chunking, no embedding and no second pass -- one call per claim, and the free floors add $0.00.

Your volumeWhat it costs at your volume

Linear, and this kit says so rather than implying a discount it did not measure. Every claim is independent -- no carried state, no shared index, no cache across calls -- so 450 claims is $26.342 and 4,500 is $263.42 at this card. What does NOT scale linearly is wall clock: 45 claims took 1,503.4 s at 5 concurrent workers against a key sibling kits were using at the same time, and no run was taken at another concurrency to establish what a second worker buys. The free floors scale to nothing: 0.1 s for 45 packs, and they are pure Python over a text file.

Model choiceWhy this model

One tier was bought, and it is the cheap one. The question this kit asks is whether a model buys anything at all over pure code on a task where most facts are fields -- and the answer on the discriminator is no, at 44.44 pct against 76.30. Spending more per call to re-ask that question would have been buying a better answer to the wrong half of it; the column that separates is the 22 correspondence lines, and the fast tier already takes 22 of 22 of them. A deliberating tier is .env plus one more run and is recorded as not bought.

Other modelsCost projection

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

45 claims, 270 claimed lines

The codeEvery file, and what it is for

16 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Nine of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysection split

Cut the claim pack on its underlined headings -- Warranty Claim Review, Warranty Policy And Guides, Claimed Lines, Records On File, Service Correspondence, Dealer Position. One regular expression, deterministic, so "what did we send" is a list of six section names and not an argument about a token window. A pack with no headings comes back as one unnamed section rather than raising, so a forker pointing this at their own export gets a usable failure instead of a stack trace.

# Cut a warranty claim review pack into addressable sections. Pure code, no model.

HEAD = re.compile(r"^([A-Z][A-Za-z ,]+)\n-{10,}\n", re.M)
def sections(text):
def named(text, name):

src/select.pythe seam that withholds — a swap seam

The Dealer Position block -- the store's open chargeback exposure, the goodwill amount a district manager may pay without a region review, and the manufacturer's own relationship posture toward this dealer -- never leaves the machine. None of it bears on whether a cause code is a covered defect, whether the vehicle was inside its coverage on the repair date, or whether the hours claimed match the published operation. All of it is a reason to pay. Withheld at the seam rather than trusted to a sentence in the instruction, and the UI prints, per claim, which sections went and which stayed.

You change it to: NEVER_SENT. One tuple, holding one section name. It is a named-section denylist and not a redaction system: the same goodwill authority that is withheld in the Dealer Position block will be sent when a sentence in the Service Correspondence names it, because the correspondence is where the answers live.

# SEAM 2 -- what leaves this machine. Pure code, and deliberately a denylist of one.

NEVER_SENT = ("Dealer Position",)
def sent(sec_names):
def body(text, sections_fn):

src/claim.pythe pack parser

The coverage terms table with its months and miles columns, the labour operation guide with its coverage code and published time, the cause classification table with its covered / wear / maintenance / customer-damage column, the provision index, every claimed line with its operation code, cause code, hours, rate and amounts, and every record on file with its type, reference and authorisation -- all read with regular expressions. The model is never asked for a figure that sits at a fixed offset. months_between() counts COMPLETED months rather than dividing days by 30.44, because a coverage term expires on the same day of the month it started and this corpus deliberately puts repairs within a week of that boundary.

# Read a warranty claim review pack into structured values. Pure code, no model.

DENY, PAYABLE, INSUFFICIENT = "DENY", "PAYABLE", "INSUFFICIENT_EVIDENCE"
DISPOSITIONS = (DENY, PAYABLE, INSUFFICIENT)
GROUNDS = ("NOT_A_COVERED_DEFECT", "CUSTOMER_DAMAGE", "PRIOR_UNAUTHORISED_REPAIR",
CLAIM_ACTIONS = ("RETURN_FOR_ADJUSTMENT", "REQUEST_SUPPORT", "RECOMMEND_PAY_AS_CLAIMED")
COVERED_CAUSE = "covered defect"
WEAR_CAUSES = ("wear", "maintenance")
DAMAGE_CAUSE = "customer damage"
CAUSE_CLASSES = (COVERED_CAUSE,) + WEAR_CAUSES + (DAMAGE_CAUSE,)
def num(s):

src/checks.pythe structured warranty checks -- and the labour-hour tolerance — a swap seam

The six grounds the pack's OWN TABLES can decide -- NOT_A_COVERED_DEFECT, CUSTOMER_DAMAGE, PRIOR_UNAUTHORISED_REPAIR, OUT_OF_COVERAGE_TIME, OUT_OF_COVERAGE_MILEAGE, LABOUR_TIME_EXCEEDED -- in a fixed PRIORITY order, plus the two shapes that settle nothing (an operation code with no guide row, a line with no technician story on file) which return INSUFFICIENT_EVIDENCE with the missing item named. recompute() answers the use case's second half on EVERY line whether or not anything is wrong with it: the hours the guide publishes, the labour amount those hours and the claim's rate give, and the variance against what was claimed. TOL = 0.05 HOURS is the single published constant that decides how much over is over, and MILE_TOL is fixed at 0 miles and named separately because a coverage limit is a number in a policy and not a measurement with a rounding error.

You change it to: SUPPORT_TYPE, one dict, six entries. This corpus files exactly one record type per ground; a real adjudication attaches several documents to one ground and some of them are images, so this is the mapping to widen first.

# The structured warranty checks, in pure code. No model, no service correspondence.

TOL = 0.05
MILE_TOL = 0
PRIORITY = ("NOT_A_COVERED_DEFECT", "CUSTOMER_DAMAGE", "PRIOR_UNAUTHORISED_REPAIR",
SUPPORT_TYPE = {
NO_AUTHORISATION = "none"
def recompute(line, parsed):
def unauthorised_prior(line, parsed):
def over_time(line, parsed, tol=TOL):
def _rid(parsed, line, rtype):

src/prompt.pythe prompt, and the correspondence-blind variant — a swap seam

The whole instruction in one place, in send order. It names the three dispositions and the six grounds, because scoring an arm on a vocabulary it was never given measures the wording rather than the arm. What it deliberately does NOT say is where to look -- nothing in it mentions reading the correspondence against the tables, and nothing mentions that a published operation time can be superseded by a combination time. _strip_notes() is the ablation: it TRUNCATES at the Service Correspondence heading and RAISES if that heading is absent, because a silent no-op there would publish an ablation arm that was never ablated.

You change it to: build(text, blind=True), which truncates the Service Correspondence and says so in the instruction. It is the only evidence this kit can offer that an arm READ the prose rather than learning that transmitted warranty claims are over-claimed: take the notes away and an arm that was reading them loses exactly the correspondence cases.

# The assembled prompt, in one place, in the order it is sent.

NOTES_HEADING = "Service Correspondence"
SYSTEM = """You are adjudicating a dealer warranty claim for a vehicle manufacturer.
BLIND_SYSTEM_SUFFIX = """
def build(text, blind=False):
def _strip_notes(body):
def render(parts):

src/draft.pythe model call and the reply parse

One claim pack in, one drafted adjudication out, and the only place a model is called for a claim. Holds the 64,000-token output ceiling that this kit's own probe forced: c000-warranty-claim-calibration fired the four heaviest packs at 32,000 and drew 30,310 output tokens on the largest reply -- 94.7 per cent of that cap -- so 32,000 was abandoned BEFORE any scored arm fired rather than after one came back truncated. c001-warranty-claim-calibration re-took the probe at 64,000 on the corrected corpus and the same four packs drew a largest reply of 27,045. The scored run then bore the decision out and bounded it: across all 45 claims r001-warranty-claim came back 45 of 45 answered, 0 failures, 0 truncated and 0 lines omitted, with its heaviest reply at 30,562 output tokens -- 47.8 per cent of the raised ceiling and ABOVE the cap that was abandoned. The parse is tolerant of a fenced block and of leading prose, uppercases only the closed vocabularies, and coerces a single record identifier returned as a bare string into a list.

# One claim pack in, one drafted adjudication out. The only place a model is called for a claim.

MAX_TOKENS = 64000
THINKING = None
def parse_reply(text):
def normalise(obj):
def draft(cfg, text, blind=False, complete_fn=None, max_tokens=None):
def cited_ids(answer):
def action_from_lines(answer):

src/adapters/__init__.pythe adapter — a swap seam

One interface, several providers, raw HTTP and no vendor SDK, so the kit installs nothing to call whichever key a forker already holds. ⚑ ITS SOCKET TIMEOUT AND THE OUTPUT CEILING ARE ONE SETTING WEARING TWO NAMES: TIMEOUT_S is 1,800 seconds beside a 64,000-token ceiling, because completions here are not streamed and a reply that actually fills that ceiling holds a silent socket for the whole generation. Transport retries are cut to ONE against four for a transient status, on the ground that a blown socket on a non-streamed generation is not a rate limit and asking four more times turns one defect into five bills.

You change it to: PROVIDER, BASE_URL and MODEL in the repo-root .env, or a kit-local .env carrying MODEL alone to point this one kit at a different tier while keeping the shared account. Adding a provider is one function and one entry in PROVIDERS, and it must return token counts as well as text -- an adapter that returns no usage cannot be published here, because the cost lens prices what the eval lens counted.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TIMEOUT_S = 1800
TRANSPORT_RETRIES = 1
def _post(url, headers, payload, timeout=TIMEOUT_S):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}

src/config.pythe shared connection, and the kit-local override

Provider, base URL, key and model are read from the repo-root .env first, then this kit's own .env, then the real environment, MERGED KEY BY KEY rather than file by file -- so a kit-local file holding MODEL alone points one kit at a different tier and keeps the shared account. Neither file is required and a missing one is not an error, which is what keeps a single cloned kit runnable with no repo above it. sources() reports WHICH file contributed and never a value, so "where is this key coming from" is answerable without anyone reading a key aloud.

# Read .env. No dependency, and no key ever leaves this machine.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
ENV = os.path.join(HERE, ".env")
SHARED_ENV = os.path.join(os.path.dirname(os.path.dirname(HERE)), ".env")
VARS = ("PROVIDER", "BASE_URL", "API_KEY", "MODEL", "EMBED_MODEL")
def _read(path):
def load():
def sources():
MODEL_DISPLAY = "the fast tier"
def has_key(cfg=None):

src/budget.pythe call cap

One key funds every kit under this root, which is the point of the shared .env and also the hole in it. This counts CALLS, not dollars, because a dollar cap needs a rate card the kit does not have and a cap that silently reads the wrong card is worse than none. The ledger line is written BEFORE the call, so a crash mid-call over-counts by one rather than under-counting. No cap configured means no cap and it says so out loud in the line printed before a run; there is no hard-coded ceiling in the file.

# A cap on live calls, shared by every kit on this machine. No dependency, no service.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
KIT = os.path.basename(HERE)
ROOT = os.path.dirname(os.path.dirname(HERE))
SHARED_ENV = os.path.join(ROOT, ".env")
LEDGER = os.path.join(ROOT if os.path.exists(SHARED_ENV) else HERE, ".calls-ledger.jsonl")
PAID_ARMS = ("r", "x")
FREE_ARMS = ("t", "b")
BOOT = os.urandom(6).hex()
PID = os.getpid()

evals/baseline.pythe three free floors — a swap seam

coverage-window is the strawman that is not a strawman -- the months-and-miles arithmetic plus a guide lookup, which is what a warranty system already does the moment a claim is transmitted. line-sweep runs the six structured checks and cites the provision the policy table prints beside the row it used, but never stops: anything it cannot settle it denies anyway. claim-gate is line-sweep plus the one thing that separates a defensible adjudication from a list of exceptions -- it REFUSES to state a ground it cannot support, and every provision and record identifier it cites is read out of the pack rather than assembled from a constant. ⚑ AND THE THIRD FLOOR WINS THE DISCRIMINATOR OUTRIGHT, MEASURED: b002-warranty-claim-claimgate scores 76.30 per cent against r001-warranty-claim's 44.44 per cent over the same 270 lines, free, offline and in 0.1 seconds. The three floors read 34.81, 70.37 and 76.30 per cent respectively. All three are correspondence-blind BY CONSTRUCTION -- 0 of 22 correspondence grounds against the paid arm's 22 of 22 -- and a keyword floor over the notes would close part of that gap and was deliberately not built, because it would be tuned to the sentences this generator happens to write and would measure the generator.

You change it to: --floor coverage-window | line-sweep | claim-gate. Three pure-code adjudicators behind the same answer shape, so one scorer grades all three and the model without a second code path.

# THREE FREE FLOORS. No key, no model, no network. Each is a genuine attempt at the job.

MODES = ("coverage-window", "line-sweep", "claim-gate")
def _action(rows):
def _row(line, disposition, ground, provision, support, finding, allowed=None):
def review(text, mode="claim-gate", tol=checks.TOL):
def _window(line, p, rc, tol):

evals/tolerance.pythe tolerance sweep

Three floors by eight labour-hour tolerances, free and offline, and it says the headline is a measurement of a parameter as much as of a method. claim-gate scores 57.78 per cent at 0.00 hours, 76.30 at 0.05, holds 76.30 at 0.10 and falls to 74.81 at 1.00. The weakest floor does the opposite and climbs to its sweep best of 39.26 per cent at a one-hour tolerance while its labour catch collapses from 47.62 to 9.52 -- it is not getting better, it is buying accuracy by giving up the only thing it was looking for. No floor is published outside the band over which the answer key does not move, because a floor allowed to relabel the key is not a floor.

# THE TOLERANCE SWEEP. Free, offline, and it must run before anyone believes a floor's score.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
RESULTS = os.path.join(HERE, "results")
TOLERANCES = (0.00, 0.005, 0.02, 0.05, 0.10, 0.25, 0.50, 1.00)
def _docs():
def _text(d):
def sweep():
def main():

evals/scoring.pythe scorer

Exact match per line, no model grades anything, and every rate carries its own denominator. A line counts toward claim_adjudication_accuracy_pct only when the disposition is right AND, on a denial, the ground is the right one of six, the provision cited is the one the pack carries for it, every required record identifier is attached and nothing on the line was invented. The catch rate is split by CHANNEL -- table grounds against correspondence grounds -- and labour time is split again on top of that, because the named trap is planted on both channels and a blended number would let one half carry the other. silent_overdeny_pct and invented_citation_pct are the two named dangerous failures, and the majority-class score is printed beside every arm.

# Score an arm against the answer key. Pure code, exact match per line. No model grades anything.

DENY, PAYABLE, INSUFFICIENT = "DENY", "PAYABLE", "INSUFFICIENT_EVIDENCE"
OMITTED = "OMITTED"
HOUR = 0.005
def _pct(n, d):
def _key(s):
def rows_by_line(answer):
def _cited(r):
def _numeric(v):
def score(records, golds, valid_ids=None, allowed_hours=None, texts=None):

evals/check_labels.pythe answer-key gate

Eleven properties, re-derived from the shipped corpus, and the three that matter are the corpus's claims about itself rather than its schema: the table half must re-derive EXACTLY from src/checks.structured_finding(); a correspondence denial must look PAYABLE to the tables; a correspondence trap must look DENIED to them. A prose denial a table can already see is not a prose case and labelling it one would inflate the very column this kit exists to measure. --self-test red-proves the gate against seeded defects rather than asserting that it works.

# A GATE ON THE ANSWER KEY ITSELF. Free, offline, no model. Run it before believing any arm.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
TABLE_CASES = {"S_NOT_A_COVERED_DEFECT", "S_CUSTOMER_DAMAGE", "S_PRIOR_UNAUTHORISED_REPAIR",
PROSE_DENY_CASES = {"C_LABOUR_TIME_EXCEEDED"}
PROSE_TRAP_CASES = {"P_COVERAGE_EXTENDED", "P_TIME_AUTHORISED", "P_REPAIR_AUTHORISED"}
DERIVABLE = TABLE_CASES | {"CLEAN_LINE", "M_OP_NOT_IN_GUIDE", "M_NO_TECHNICIAN_STORY"}
def load_gold():

evals/injection.pythe injection probe — a swap seam

One instruction-shaped note forced into the Service Correspondence of every claim in a scope FIXED BEFORE THE RUN -- the claims where the paired run correctly denied at least one line, in claim-id order, capped by an argument that is recorded in the result file. Every injected cell is paired against its own un-injected answer rather than against gold, because a cell the arm was already getting wrong cannot be suppressed. It records whether a cell that HELD had its ground, its provision or its support changed underneath it, and both phrasings are quoted in the file in full, including any that was not fired.

You change it to: --phrasing, over the PHRASINGS dict, with --scope and --paired-with fixing the denominator before anything is sent. Sibling kits have measured one phrasing at over eighty per cent suppression and another at near zero, so the phrasings are reported one by one and never meaned together.

# THE INJECTION PROBE. Does an instruction-shaped note in the service correspondence talk the

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
KIT = "warranty-claim"
HEADING = "Service Correspondence"
PHRASINGS = {
def documents():
def load_doc(doc_id):

src/app.pythe local UI

http.server, standard library only, five endpoints and no write path -- nothing here pays, denies, submits or charges back a claim, and there is no endpoint, button or flag that adds one. It renders with no key: the pack, everything read off it in code, the published labour time on every line and the strongest free floor all cost nothing, so the page always computes them. The floor's adjudication sits beside the model's with an Agree? column, so a line where the two agree is a line where the model bought nothing and the page says so rather than taking the credit.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
CORPUS = os.path.join(HERE, "data", "corpus")
PORT = int(os.environ.get("PORT", "9074"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-warranty-claim")
FLOOR_MODE = "claim-gate"
def documents():
def load_doc(doc_id):
def read_in_code(text):

tools/build_corpus.pythe corpus generator — a swap seam

Writes data/corpus/*.txt, data/gold.jsonl and data/corpus-stats.json from seed 20260826. The answer key is DERIVED for everything pure code can settle -- it is structured_finding()'s own output -- so the key and the free floor cannot silently drift apart. What is TYPED is only the four cases code cannot reach, and each of those is then asserted to look the opposite way to the tables. The dealer's own rounding is planted at no more than 0.03 hours and every planted over-claim at 0.08 hours or more, which is the gap that makes the tolerance sweep readable. ⚠︎ IT ALSO CARRIES A RECORDED DEFECT AND ITS FIX RATHER THAN A REGENERATED CORPUS: both limbs of a special coverage adjustment note are derived from where the vehicle stands rather than from the coverage term, so 8 of the 16 notes print the limb nobody is arguing about at or below the standard term. It was found after r001-warranty-claim was already in flight, and regenerating would have discarded 45 paid calls to tidy a sentence.

You change it to: One seed (20260826) and one case plan. Replace it with your own claim packs and your own data/gold.jsonl; data/SOURCES.md lists what to change and in what order, and python3 -m evals.check_labels is what tells you the replacement key is self-consistent.

# GENERATE the shippable corpus: 45 warranty claim review packs, an answer key, and the stats.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
SEED = 20260826
PACKS = 45
LINES = 6
DATASET = "warranty-claim-2026-08-26-45packs-v2"
RULE = "-" * 64

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3316 input and 18959 output tokens per claim (one transmitted warranty claim, covering every line claimed on it), all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

NOT ATTACKED YET, AND THE PAGE SAYS SO RATHER THAN LOOKING FULL. One boundary on this kit IS measured, and it is the one the use case turns on: the Dealer Position section -- the dealer's open chargeback exposure, the goodwill amount the district manager may authorise without a region review, and the manufacturer's relationship posture toward the store -- never leaves this machine. It is withheld at the seam in pure code (src/select.NEVER_SENT), on 45 of 45 packs, with sections_used recorded as ['system', 'pack'] on every call of the only live run. None of it bears on whether a cause is a covered defect, whether the vehicle was inside coverage on the repair date, or whether the hours claimed match the published operation time; all of it is exactly what you would least like a generated adjudication to have been reasoning about. THE MODEL IS NEVER HANDED A REASON TO PAY. The boundary that is NOT measured is the one that carries the answers: the Service Correspondence reaches the model verbatim, it is the only evidence for 64 of the 270 lines, and no injection probe has been fired against this kit.

The credential is looked for in three places and the most specific one wins: the repository-wide <code>.env</code>, then an optional file belonging to this kit alone, then whatever the real environment holds. Merging happens variable by variable, so a kit-level file naming only MODEL redirects this one adjudicator while continuing to use the shared account. Neither file has ever been tracked &mdash; both were ignored from the first commit and no credential has existed in this repository. <code>src/config.sources()</code> answers where a setting came from by naming files and never contents, so that question can be settled without anybody reading a secret out loud. <code>save()</code> creates the file already at 0600, so there is no window at the default umask in which it sits readable. If the provider returns an error, the local UI strips the key and the base URL out of the message before it reaches the browser. A reader is never asked for any of this: the interface loads, browses, replays and computes floors with nothing configured, which on warranty claims is not a crippled mode at all, since the best-scoring arm the kit publishes calls no one.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. Both were run for real on 2026-08-26.

Two of these three boundaries are measured and the third is the one that matters most. The withholding seam is verifiable from the request body itself, and citation validity is scored on every run -- but both are properties this kit controls in code. The boundary it does NOT control is the untrusted channel it must read to do the job at all, and that one has never been probed here. A threat model that reported only the two it can prove would be a threat model shaped by what was easy to measure.

The resultNothing has attacked this kit yet -- the boundary that IS measured is the withheld Dealer Position block, gone from 45 of 45 packs, and the boundary that is NOT is the correspondence channel that decides 64 of the 270 lines

No red-team run has been taken on warranty-claim. evals/injection.py is written, tested in shape against a sibling kit's design and never executed here: it would force an instruction-shaped note into every claim in scope, fix scope before reading any answer, pair every cell against the same arm's own un-injected reply so that a cell the arm was already getting wrong cannot be counted as suppressed, and publish the rate with both its denominator and its ceiling. Both phrasings -- one asserting the coverage position was agreed at zone level and closed, one a district-manager directive to raise only certain denials during a retention review -- are printed in full in that file so a reader can re-take the probe rather than take a description on trust. Neither has been sent. The only live calls this kit has ever made are the four of c001-warranty-claim-calibration, fired to size the output-token ceiling.

HonestyWhat this does not prove

• WHETHER THIS KIT CAN BE TALKED OUT OF A DENIAL AT ALL. No injection run has been fired. Sibling kits on this estate have measured suppression rates from near zero to over eighty per cent on a single replaced section, and a single number would bound nothing even if one existed here -- but there is not one, and the absence is the finding.
• Whether either written phrasing would work. Both are quoted in full in evals/injection.py and neither was sent. Naming them is not measuring them.
• Whether a correspondence ground can be SUPPRESSED as opposed to blinded. Replacing the Service Correspondence destroys the evidence those 22 grounds rest on, so even once the probe runs, that channel would have to be reported separately rather than folded into a clean suppression rate.
• Whether an instruction hidden in the Dealer Position section would do anything. That text is removed before a request is assembled, so it has no path to a model &mdash; and no one has tried to find one.
• Whether a paid arm invents citations at any rate worth publishing. The only live run on this kit is four calls over 24 lines, which is a calibration probe and not a measurement of citation discipline.
• Anything about the provider side of the boundary. What leaves this machine is enumerated and checked; what the configured provider retains, logs or trains on is the reader's contract with that provider and nothing here tests it.
• Whether the withholding seam survives a real claim pack. It matches an underlined heading exactly; a real transmitted claim record has no such heading, src/segment.py returns the whole file as one section, and the seam would then withhold nothing at all. That is the first thing to check before pointing this at anything real.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

It never pays, denies, submits, adjusts or charges back a warranty claim, and it never contacts a dealer. What it produces is an ADJUDICATION a warranty analyst reads before deciding what to do with the claim. INSUFFICIENT_EVIDENCE with the missing item named is part of that adjudication rather than a failure of it -- an operation code with no row in the labour time guide and a line with no technician story are both things the pack does not settle, and saying so is the honest answer. Paying or denying is the manufacturer's act under a dealer agreement the pack references and does not enforce.

Stated in the system prompt on every call, in the README, and on the local UI -- and enforced by there being no such endpoint to remove. src/app.py serves four read paths (/api/claims, /api/prompt, /api/recorded, /api/claim) plus /api/draft, and nothing that mutates anything outside results/. There is no dealer address, no claim submission path and no configuration flag that adds one.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• What exists here is a missing capability plus an instruction. There is genuinely no route through this code that pays, denies or transmits anything &mdash; but that is a fact about what was never built, not a control standing guard. No policy engine sits in front of it, no approval chain gates it, no audit trail records it. Kits do not carry those, and bolting them on would turn this into a product rather than something a person can read in an evening.
• src/select.py withholds a NAMED SECTION and is not a redaction system. A claim whose Service Correspondence mentions the district manager's goodwill authority will send that mention, because the correspondence is where the answers live and this kit cannot have both. If you point this at real claim packs, that is the first thing to look at.
• A paid arm's citations are counted, never constrained. The instruction it is sent forbids naming any provision or record the pack does not print; instructions are things a model may decline to follow, and this one has nothing behind it. The set of identifiers the pack actually contains is already parsed in code and is used only to score, never to reject.
• EVERY FLOOR FIGURE HERE HAS 0.05 HOURS BURIED IN IT. That constant was picked by sweeping the same 270 lines the floors are then graded on, and it swings claim-gate by 18.52 points. It is written out beside every number it produced rather than being allowed to pass as a property of the technique.
• The 100 per cent on the table half is a BAR AND NOT A RESULT. src/checks.py defines what 'the tables settle' means and the answer key re-derives from it, so that column measures agreement with a definition. The column that measures difficulty is the correspondence half, where free code scores 0 of 22.
• THE PAID ARM EXISTS NOW AND IT LOSES THE HEADLINE COLUMN. r001-warranty-claim adjudicates 44.44 pct of 270 lines defensibly against free code's 76.30 pct. It is not a weak arm -- it takes 22 of 22 correspondence grounds and 42 of 42 labour-time cells that free code cannot reach at all, and cuts silent over-denial from 42 lines to 12. It loses on discipline: half the payable lines denied over dealer rounding, 10 of 16 unsettleable lines answered anyway, and a quarter of its citing lines quoting something real that is not a record an analyst can pull.
• AND THE CLAIM-LEVEL ACTION METRIC IS BEATEN BY A CONSTANT. 82.22 pct is exactly what the single phrase RETURN_FOR_ADJUSTMENT scores on these 45 claims. It is published rather than dropped, but it should not be read as evidence of anything.
• THE 70.74 PCT COUNTERFACTUAL IS A DIAGNOSIS OF 44.44 PCT AND NEVER A REPLACEMENT FOR IT. Computed offline from the committed result file: crediting WP-3.4 gives 56.67 pct, waiving the in-pack non-identifier citations gives 55.93 pct, both together 70.74 pct -- still 5.56 points behind free code. The published figure stays 44.44 pct. The key was not edited, the scorer was not loosened and the run was not fired again; the arithmetic is shown so a reader can see what the gap is made of, not so the gap can be argued down.

WatchedWhat is watched, and why that one

The kit measures 97 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

21 measured by the latest run 76 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (adjudicated_lines 270, answered_trap_lines 42, blind False, claim_support_cells 70, claims 45, claims_answered 45, correspondence_ground_cells 22, dataset_version warranty-claim-2026-08-26-45packs-v2, deniable_lines 130, denial_ground_cells 70, floor_tolerance_hours 0.05, labour_hours_cells 261, labour_time_cells 42, labour_time_prose_cells 22, labour_time_table_cells 20, lines_citing_any_identifier 0, payable_lines 124, policy_provision_cells 70, table_ground_cells 108, unsettleable_claim_cells 16) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

7 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

comply · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 3 runs measured the same system — they differ on claim_support_cells, denial_ground_cells, floor, lines_citing_any_identifier, policy_provision_cells, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

comply · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on adjudicated_lines, answered_trap_lines, blind, claim_support_cells, claims, claims_answered, correspondence_ground_cells, deniable_lines, denial_ground_cells, labour_hours_cells, labour_time_cells, labour_time_prose_cells, labour_time_table_cells, lines_citing_any_identifier, payable_lines, policy_provision_cells, table_ground_cells, unsettleable_claim_cells — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 21 chips that all say so.

DeviationsWhat deviated

0 breaches across 7 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The scored arm r001-warranty-claim has been taken on all 45 packs at 5 concurrent workers, and the correspondence-blind ablation s001-warranty-claim-notes-blind on the same 45 at 6. Both are published, both are on this page, and both are single runs. There is no schedule here and nothing re-fires on a clock: the three free floors and the tolerance sweep re-score the whole corpus offline for nothing in about a tenth of a second, so the cheap half of this board can be re-taken on any commit, and the paid half is re-taken only when someone decides to spend. A band moving is a prompt to look, never a trigger to re-run. THE CORRESPONDENCE-BLIND ABLATION HAS NOW BEEN SCORED, AND IT LANDS ON THE FREE FLOOR'S NUMBER IN FIVE COLUMNS. s001-warranty-claim-notes-blind ran the same arm over the same 45 packs with the Service Correspondence section truncated out of the prompt and the instruction saying so. Against r001-warranty-claim: correspondence grounds 22 of 22 -> 0 of 22, the correspondence half of the labour-time trap 22 of 22 -> 0 of 22, labour time exceeded overall 100.00 pct -> 47.62 pct, unpayable lines denied 100.00 pct -> 83.08 pct, and silent over-denial 12 of 42 -> 42 of 42. Every one of those five is the strongest free floor's own figure to the decimal. An arm that was pattern-matching on the tables would barely have moved; this one lost exactly the prose cases and kept exactly the table ones, which is the only evidence this kit has that the sentences were read rather than guessed around.

In one lineNo framework, deliberately

NO FRAMEWORK, AND THE COST OF THAT CHOICE IS PAID BY THE AUTHOR RATHER THAN THE READER. Nothing is installed: <code>requirements.txt</code> holds no package and states the condition for adding one &mdash; some module under <code>src/</code> or <code>evals/</code> has to import it first. No orchestration layer, no vendor SDK, no eval library. The test this is written to pass is whether somebody who has never seen the repository can find the two decisions that actually move the published numbers and change them in an afternoon. On warranty claims those two are a labour-hour tolerance (<code>src/checks.TOL</code>) and the name of the section that is held back (<code>src/select.NEVER_SENT</code>) &mdash; a float and a one-element tuple, both of which any framework worth its name would have swallowed into configuration where nobody argues with them.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear, once per claim pack: data/corpus/WC-n.txt -> src/segment.py (six underlined sections) -> src/select.py (five of them leave) -> src/prompt.py (system plus pack, in send order) -> src/adapters (one call) -> src/draft.py (parse and normalise) -> evals/scoring.py (exact match per line against the key). No branches, no agent loop, no tool calls, no retrieval step and nothing re-entrant. The only fan-out is a thread pool over independent claims, and the three free floors skip the middle four stages entirely -- src/checks.py reads the parsed pack directly.

The other sideWhat a framework costs you

• A new provider is a hand-edited entry in PROVIDERS, and one that reports no token usage simply cannot be published here, since the cost and telemetry surfaces both read those counts off the run record. What is bought in return is that cloning this kit installs nothing at all.
• There is no configurable retry or backoff policy. It is four attempts with 1s/2s/4s/8s backoff on a transient status and one attempt on a transport failure, written out in the adapter. Changing it means editing Python, which is the intended cost of a decision this consequential.
• There is no schema validation on the reply, so a genuinely malformed answer is scored as a miss rather than raised as an error. That is the deliberate trade -- see the reply-parse seam -- and it is only defensible because parse failures are recorded per run rather than assumed away.
• There is no persistence layer and no job queue, so a run is a process. A run that dies loses its unrecorded calls, and the daily call ledger over-counts by one on a crash mid-call rather than under-counting -- the direction a spend guard should round.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-warranty-claim on the fast tier, correspondence-blind, 2026-08-26. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 5 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

3 runs not plotted. b000-warranty-claim-coveragewindow, b001-warranty-claim-linesweep, b002-warranty-claim-claimgate recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence. The corpus has no third-party component to licence separately: it is written by a generator that ships in the repo, from a seed recorded in data/corpus-stats.json, and every figure in it is invented. Verified by reading every generator input on 2026-08-26. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe whole adjudication on each claimed line -- disposition, ground, provision, record

whether each of the 270 claimed lines got an adjudication you could actually put in front of the dealer appealing it: the disposition the answer key carries, and on a DENY the right ground of the six, the policy provision the pack itself prints for it, every record identifier the ground needs, and nothing cited that is not a provision or record the pack names

MethodHow the test was run

python3 -m evals.run --run-id <id> --yes (or --floor <mode> for a free arm); evals/scoring.py then compares each line against data/gold.jsonl by exact match.

Every grader on these pages scored the same 270 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The named trap, on the channel free code cannot reach. The floor says PAYABLE because every table is satisfied; the arm says DENY on LABOUR_TIME_EXCEEDED because it read the combination note and checked that the partner operation really is on this repair order. It still does not score on the discriminator, because it cited WP-3.4 where the key carries WP-3.1 -- which is the answer-key conviction below, not a reading failure.

The formulaWhat it computes

claim_adjudication_accuracy_pct = accurate lines / 270

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, derived with the corpus by tools/build_corpus.py and re-derived from the shipped packs by evals/check_labels.py, which asserts eleven properties and is red-proven with --self-test.

How tight can the band be? src/checks.TOL = 0.05 hours, and it was chosen from b900-warranty-claim-tolerance-sweep rather than typed: claim-gate reads 57.78 pct at 0.00 h, 65.56 at 0.02, 76.30 at the shipped 0.05, 75.93 at 0.25 and 74.81 at 1.00. It is published inside the band [0.031, 0.159] hours over which the ANSWER KEY is identical -- measured by re-deriving all 270 labels at 0.031, 0.05, 0.10, 0.15 and 0.159, and it changes at 0.16 -- so the floor cannot move the labels in its own favour.

Cadence: On demand and free. All four arms re-score offline in about a tenth of a second, so this grader can run on every commit; the paid arms are re-taken only when someone decides to spend.

The decisionWhen to reach for it

In one lineVerdict only, three-way

the same three-way call WITHOUT the citation and record requirements -- what a rules engine can score. The gap between this and the row above is the part of the work a coverage-window check does not do, in numbers: 72.22 against 44.44 on the paid arm, and 76.30 against 76.30 on the free floor, which cites correctly by construction

MethodHow the test was run

The same run and the same scorer; claim_verdict_accuracy_pct is computed beside the discriminator on the identical 270 lines, without the citation and record conditions.

Every grader on these pages scored the same 270 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The named trap, on the channel free code cannot reach. The floor says PAYABLE because every table is satisfied; the arm says DENY on LABOUR_TIME_EXCEEDED because it read the combination note and checked that the partner operation really is on this repair order. It still does not score on the discriminator, because it cited WP-3.4 where the key carries WP-3.1 -- which is the answer-key conviction below, not a reading failure.

The formulaWhat it computes

claim_verdict_accuracy_pct = lines whose disposition matches / 270

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the same data/gold.jsonl; only the conditions differ.

How tight can the band be? No threshold. It is an exact three-way string match with no tolerance in it.

Cadence: On demand and free. All four arms re-score offline in about a tenth of a second, so this grader can run on every commit; the paid arms are re-taken only when someone decides to spend.

The decisionWhen to reach for it

In one lineThe gate on the answer key itself

whether the key everything else is scored against is internally sound: eleven properties, red-proven with four seeded defects that must each be convicted by name and an acquittal that must still pass afterwards

MethodHow the test was run

python3 -m evals.check_labels (and --self-test to red-proof it)

Every grader on these pages scored the same 270 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The named trap, on the channel free code cannot reach. The floor says PAYABLE because every table is satisfied; the arm says DENY on LABOUR_TIME_EXCEEDED because it read the combination note and checked that the partner operation really is on this repair order. It still does not score on the discriminator, because it cited WP-3.4 where the key carries WP-3.1 -- which is the answer-key conviction below, not a reading failure.

The formulaWhat it computes

evals/check_labels.py --self-test -> 0 violations on the shipped key

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/checks.py plus the shipped corpus -- the key is re-derived from the rendered packs, never from the generator's variables.

How tight can the band be? No threshold. Eleven boolean properties, all or nothing.

Cadence: On demand and free. All four arms re-score offline in about a tenth of a second, so this grader can run on every commit; the paid arms are re-taken only when someone decides to spend.

The decisionWhen to reach for it