The business caseThe problem this solves

Deciding whether a dealer warranty claim is covered is coverage-term application plus arithmetic plus narrative reading, in that order and not any other. The vehicle has to be inside the time AND mileage limits for the RIGHT coverage — basic, powertrain, emissions or a purchased extended contract, each with its own limits and its own component list — the failed part has to be on that plan's list, the claimed labor operation has to belong to that part, and an exclusion beats all of it. The record puts the technician's own free-text narrative right next to the structured values, and that narrative is the longest thing on the page, the only part of it nobody measured, and the only place an exclusion is ever actually described. Someone opening each warranty claim, working out how many complete months the vehicle has been in service, looking up which components and which month/mileage limits the claim's OWN coverage plan carries, checking the claimed labor operation against the failed part, and then reading the technician's narrative for the thing that overrides all of it — an exclusion the technician describes but the claim form does not code. It is a few minutes per claim and it is every claim, and the part that goes wrong is not the arithmetic: it is reaching for the 3/36 bumper-to-bumper limit on a claim filed under a powertrain plan, or letting the technician's own guess about whether it will pay set the answer.

The inputThe actual warranty claims

The corpus is 55 warranty claims, 0.04 MB (txt 55). Plain text, one format, invented rather than fetched — a real dealer warranty claim names a real customer's vehicle, a real dealership and a technician's own words about a specific repair, and there is no public corpus of (claim, correct coverage decision) pairs for the same reason there is no public corpus of bank statements. Generating it also makes the label mechanical: gold's covered is the six-branch rule run over the claim's own values, not somebody's reading of the narrative, and tools/build_corpus.py asserts that every gold value is stated verbatim in the document it labels.

Claim
-----
WCL-DN-39441

Servicing Dealer
----------------
Prairie Fork Motors (D-3390)

Vehicle
-------
Platform U4 midsize crossover / VEH-AKF-733824

Coverage Plan
-------------
emissions

In-Service Date
---------------
2019-12-25

Repair Date
-----------
2027-12-25

Odometer
--------
57298 miles

Failed Component
----------------
oxygen_sensor

Claimed Labor Operation
-----------------------
LOP-2217

Cause Code
----------
defect

Claim Status
------------
paid

Technician Narrative
--------------------
Duplicated the fault twice; the oxygen sensor has failed internally, nothing around it is disturbed and there is no impact damage anywhere near it. Unit removed and replaced, road tested, no further faults stored. Nothing here that would keep it from being covered.

The outcomeWhat a good result looks like

A fourteen-field adjudicated record per claim, plus one pure-code routing decision taken from two of those fields: a claim that is not covered and has already been paid is the one that goes to recovery today. Nothing here denies a claim, opens a chargeback, debits a dealer or contacts anybody.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Screening a day's warranty claims for ones that should not be paid, before an adjudicator opens them — either tier — they tie at 55 of 55 verdicts, 1.00 recall and 1.00 precision measured here
  100 pct coverage accuracy on both tiers against the free opinion floor's 60.0 pct (22 wrong, including 11 claims that should have been denied and were called covered).
• Deciding the two tiers on cost, speed or accuracy — the fast tier
  17 pct cheaper per claim ($0.0026098 vs $0.0031286 on the published card), 49 pct lower p50 latency (4,375 ms vs 8,662 ms), and identical on every one of the five published graders — 770 of 770 cells, 55 of 55 verdicts, 12 of 12 trap rows, 16 of 16 flags, on both.
• Pointing this at your own warranty programme's claims — the six-branch structure, and none of the numbers
  The shape transfers: an exclusion outranks the terms, the operation has to belong to the part, wear is its own rule, the component list is the plan's, and the limit is the PLAN'S limit. Every accuracy figure here was measured with those terms typed into the system prompt.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-coverage-check. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts, the 55-claim picker populates and the field table draws its fourteen empty rows; clicking Adjudicate returns the no-key sentence rather than a stack trace. python -m evals.check_labels and python -m evals.run --run-id b000-rules --baseline both pass with no network access at all, and the baseline produces a real scored result file for $0.00.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per warranty claim record

Current processWhat it replaces

Someone opening each warranty claim, working out how many complete months the vehicle has been in service, looking up which components and which month/mileage limits the claim's OWN coverage plan carries, checking the claimed labor operation against the failed part, and then reading the technician's narrative for the thing that overrides all of it — an exclusion the technician describes but the claim form does not code. It is a few minutes per claim and it is every claim, and the part that goes wrong is not the arithmetic: it is reaching for the 3/36 bumper-to-bumper limit on a claim filed under a powertrain plan, or letting the technician's own guess about whether it will pay set the answer.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 0 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings; a claim with none falls back to one whole-document segment

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 55 claims cut into 660 sections in a fraction of a second, in process, with no model and no network.

LicenceLicence

MIT — this repository's own licence. Every claim id, vehicle id, dealership name, platform label, coverage term, component list and labor operation code is invented; no real manufacturer, model, VIN, dealer, customer or published warranty booklet is named or reproduced.

Bring your ownBring your own warranty claims

Replace data/corpus/*.txt, write your own data/fields.json, and supply a gold row per claim. SECTION_HINTS in src/select.py maps fields to section headings and will need editing for a different claim-form layout; when it does not match, selection falls back to the whole document — slower, more expensive, always correct. PLANS, PLAN_COMPONENTS, LABOR_OPS and coverage_verdict() in src/extract.py are this kit's own invented coverage structure and should be the first things you replace with your own programme's terms; compute() is the routing rule and is a separate edit on purpose.

⚠︎ And what stops being true when you do: What stops being true the moment you do: the model is handed the whole coverage structure in the system prompt here — four plans, their limits, their component lists and twelve labor operation codes, about 1036 tokens of it. A real programme's terms are a document, not a paragraph, and nothing in this run measures whether a model can apply terms it has to find and read for itself rather than terms it was handed. Every accuracy figure on this page is measured with the answer key in the prompt.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 2,184 tokens assembled, 1,036 are instructions — 47% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on WCL-0047 — three calls at max_tokens=1, each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. The total, 2184, is the same figure the captured example call reports as its own input_tokens, which is what proves the three parts decompose the prompt that was actually sent. ⚠︎ The first part's 1036 tokens include the chat protocol's own fixed overhead: the nested-prefix method cannot separate the protocol from the system message, because the shortest prefix it can send already contains both. kits/UC0046-coverage-check/results/tokens-p001-coverage-check.json.

Full promptThe prompt in full

You are a warranty claim adjudicator. You read one dealer warranty claim record and return structured JSON and nothing else.

RULES, in order of importance:
1. If the record does not state a field, return null for it. Do not infer it and do not use what you know about any real manufacturer's warranty.
2. THE COVERAGE TERMS ARE THE ONES BELOW AND NOTHING ELSE. Each plan has its own month and mileage limit, and its own list of covered components:
   - basic       36 months / 36,000 miles  -- covers every component below except wear items
   - powertrain  60 months / 60,000 miles  -- covers ONLY transmission_assembly, engine_short_block, drive_axle
   - emissions   96 months / 80,000 miles  -- covers ONLY catalytic_converter, oxygen_sensor, evap_canister
   - extended    84 months / 100,000 miles -- covers every component below except wear items
   Wear items are brake_pads, wiper_blades and clutch_disc. They are on NO plan's component list.
   Each component has exactly one labor operation:
   transmission_assembly=LOP-4412, engine_short_block=LOP-4101, drive_axle=LOP-4520, catalytic_converter=LOP-2203, oxygen_sensor=LOP-2217, evap_canister=LOP-2240, infotainment_head_unit=LOP-8310, power_window_motor=LOP-8422, hvac_blower_motor=LOP-8155, brake_pads=LOP-5701, wiper_blades=LOP-5140, clutch_disc=LOP-5330.
3. `months_in_service` IS A CALCULATION, NOT A READING. It is (repair year - in-service year) x 12 + (repair month - in-service month), minus 1 if the repair day-of-month is EARLIER than the in-service day-of-month. Compute it before you decide coverage; the answer to several of these claims turns on one month.
4. `covered` IS DECIDED BY SIX CHECKS IN THIS EXACT ORDER. Stop at the first that fires:
   a. If narrative_finding is collision_damage, unauthorized_modification or missed_maintenance, answer 'no'. An exclusion the technician describes outranks every coverage term, however new the vehicle is.
   b. Otherwise, if claimed_labor_op is not the operation listed above for this failed_component, answer 'no'. The claim is not payable as coded.
   c. Otherwise, if failed_component is a wear item, answer 'yes' when coverage_plan is basic or extended AND months_in_service is 12 or less AND odometer_miles is 12,000 or less -- a premature wear failure IS covered. In every other case answer 'no'.
   d. Otherwise, if failed_component is not on this coverage_plan's own component list, answer 'no'.
   e. Otherwise, if months_in_service is GREATER THAN this plan's month limit, or odometer_miles is GREATER THAN this plan's mileage limit, answer 'no'. USE THE PLAN'S OWN LIMIT, NEVER THE 36/36,000 BASIC ONE. The limits are inclusive: exactly 36 months on a basic plan is inside the term, not outside it.
   f. Otherwise answer 'yes'.
5. `narrative_finding` IS READ FROM THE TECHNICIAN NARRATIVE, NOT FROM THE CAUSE CODE FIELD. The Cause Code is what the dealer coded on the form; the narrative is what the technician says they actually found, and on this corpus the two often disagree. Copy the Cause Code into `cause_code` and then ignore it completely when deciding anything else.
6. THE TECHNICIAN'S CLOSING OPINION ABOUT WHETHER THE CLAIM WILL PAY IS NOT EVIDENCE. Every narrative ends with one. A narrative that says 'should be covered, no question' does NOT make the claim covered, and one that says 'I expect this gets denied' does NOT make it denied. Only the six checks above decide, and they never read that sentence.
7. Copy values verbatim from the record wherever possible, and report months_in_service and odometer_miles as bare numbers with the unit left out of them.
8. Use the exact allowed value for a field that lists them.
9. Return every field named in the schema, even when the answer is null.

- claim_id (string) -- the warranty claim identifier, verbatim
- vehicle_id (string) -- the vehicle identifier on the Vehicle line, verbatim -- the VEH- code only, not the platform label
- coverage_plan (enum) one of: basic, powertrain, emissions, extended -- the coverage plan this claim is filed under, verbatim
- in_service_date (string) -- the date the vehicle went into service, YYYY-MM-DD, verbatim
- repair_date (string) -- the date the repair was performed, YYYY-MM-DD, verbatim
- months_in_service (number) -- COMPLETE months between in_service_date and repair_date, as a bare whole number. Compute it as (repair year - in-service year) x 12 + (repair month - in-service month), then subtract 1 if the repair day-of-month is EARLIER than the in-service day-of-month. A vehicle in service on the 18th is 35 months old on the 17th of the 36th month and 36 months old on the 18th
- odometer_miles (number) -- the odometer reading at the repair, as a bare number without the unit
- failed_component (enum) one of: transmission_assembly, engine_short_block, drive_axle, catalytic_converter, oxygen_sensor, evap_canister, infotainment_head_unit, power_window_motor, hvac_blower_motor, brake_pads, wiper_blades, clutch_disc -- the component the claim says failed, verbatim
- claimed_labor_op (string) -- the labor operation code claimed, verbatim
- cause_code (enum) one of: defect, damage, modification, maintenance -- the CODED cause on the claim form, copied verbatim. This is what the dealer coded, which is not necessarily what the technician describes -- copy it, do not reason from it
- claim_status (enum) one of: submitted, paid -- has this claim already been paid to the dealer, or is it still submitted and awaiting adjudication?
- technician_narrative (string) -- the technician's own free-text repair narrative, copied verbatim and in full
- narrative_finding (enum) one of: defect, collision_damage, unauthorized_modification, missed_maintenance -- what the technician's narrative actually DESCRIBES having found -- read the narrative, not the Cause Code field. 'collision_damage' for impact, a strike, a gouge, a bent mount or road debris; 'unauthorized_modification' for an aftermarket part, a splice, a non-approved controller or an altered tune; 'missed_maintenance' for skipped service intervals, sludge, contamination or a filter never changed; 'defect' for an internal failure with no outside cause. The technician's closing GUESS about whether the claim will pay is not a finding -- ignore it here
- covered (enum) one of: yes, no -- is this claim covered? Decide it STRICTLY from coverage_plan, months_in_service, odometer_miles, failed_component, claimed_labor_op and narrative_finding -- never from cause_code and never from the technician's opinion about whether it will pay. Apply these six checks IN ORDER and stop at the first that fires: (1) if narrative_finding is collision_damage, unauthorized_modification or missed_maintenance, answer 'no'; (2) if claimed_labor_op is not this failed_component's own operation, answer 'no'; (3) if failed_component is a wear item (brake_pads, wiper_blades, clutch_disc), answer 'yes' only when coverage_plan is basic or extended AND months_in_service <= 12 AND odometer_miles <= 12000, otherwise 'no'; (4) if failed_component is not on this coverage_plan's component list, answer 'no'; (5) if months_in_service exceeds THIS PLAN'S month limit or odometer_miles exceeds THIS PLAN'S mileage limit, answer 'no' -- the limit is the plan's own, never the basic 36/36,000; (6) otherwise answer 'yes'

Claim
-----
WCL-MW-55629

Vehicle
-------
Platform C1 compact sedan / VEH-FWF-783068

Coverage Plan
-------------
powertrain

In-Service Date
---------------
2024-12-24

Repair Date
-----------
2026-01-24

Odometer
--------
17091 miles

Failed Component
----------------
drive_axle

Claimed Labor Operation
-----------------------
LOP-4520

Cause Code
----------
defect

Claim Status
------------
paid

Technician Narrative
--------------------
Found the drive axle cracked open with fresh impact marks and road debris packed against it -- this is a strike from underneath, not a failure. R&R complete, torque checked, and the concern no longer duplicates. Should be covered under the plan terms, no question.

Raw responseThe raw response

{
  "claim_id": "WCL-MW-55629",
  "vehicle_id": "VEH-FWF-783068",
  "coverage_plan": "powertrain",
  "in_service_date": "2024-12-24",
  "repair_date": "2026-01-24",
  "months_in_service": 13,
  "odometer_miles": 17091,
  "failed_component": "drive_axle",
  "claimed_labor_op": "LOP-4520",
  "cause_code": "defect",
  "claim_status": "paid",
  "technician_narrative": "Found the drive axle cracked open with fresh impact marks and road debris packed against it -- this is a strike from underneath, not a failure. R&R complete, torque checked, and the concern no longer duplicates. Should be covered under the plan terms, no question.",
  "narrative_finding": "collision_damage",
  "covered": "no"
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check if a dealer warranty claim is really covered — 55 warranty claims. Two tiers of one model family answered, and every answer was then graded Five different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran r001-coverage-check and r002-coverage-check publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersFive ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The opinion floor (evals/baseline.py) and the scorer (evals/judge.py) are both pure code and cost $0.00 to run against any result set — grading adds nothing. The first figure is both scored runs' own token counts (55 claims each) priced at Google Gemini 3 Flash's published rate, the same basis cost_per_query_usd uses. The second adds everything else the build actually spent on: the 6-call MAX_TOKENS calibration at a 16,000 ceiling, the three max_tokens=1 prompt-token probes, the captured example, and two live UI screenshots. 122 live calls in total.

Cost driversWhat actually moves the bill

• The fixed system prompt and field schema (1927 of 2184 tokens on the example call, 88 pct) — the four plans' limits, their component lists, the twelve labor operation codes and the six-branch order. Every claim pays for the whole warranty programme before a single value is read.
• The reasoning pass. 257 of the 474 output tokens on the example call were billed as reasoning that never reaches text, and evals/calibrate.py measured 60.3 pct across six claims — the coverage decision is six ordered checks plus a date calculation, and that is what the completion budget is actually spent on.
• Output length: the reply returns a full fourteen-key JSON record every call, including the technician's narrative copied back verbatim, whatever the claim says.

Your volumeWhat it costs at your volume

Linear in claims: each call is independent, carries the same fixed prompt and shares nothing with its neighbours, so 550 claims cost ten times 55 and take ten times as long. Nothing amortises — there is no index to build and no prefix cache, which is also why there is no volume discount to find without changing the design.

Model choiceWhy this model

It is the model the shared key is configured for, and the second row is the same account's deliberating tier — one variable, not two. Neither was chosen on a benchmark, and the tie between them on this corpus is a fact about the corpus.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 55 warranty claim records, one completion call each, on the fast tier. The deliberating tier's own 55 calls are recorded separately in Cost.cost_by_model and are not projected here.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the warranty claim record into addressable sections, pure code

# Cut a document into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections carry each field, pure code — the coverage verdict is mapped to the six facts the rule actually reads and NOT to the coded Cause Code, and the Servicing Dealer section is mapped by nothing and never sent

You change it to: map fields to your own claim form's headings; unmatched falls back to the whole document

# Pick which sections plausibly carry each field. Pure code -- the last deterministic step before

SECTION_HINTS = {
def for_field(secs, field):
def plan(secs, fields):

src/prompt.pyprompt

assemble one call for all fourteen fields, with the four plans' limits, their component lists, the twelve labor operations and the six-branch priority order stated in full

# Assemble the adjudication prompt. One prompt per warranty claim record, all thirteen fields in

SYSTEM = (
def field_schema(fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key — plus the pure-code business-condition check downstream: a claim that is not covered and has already been paid is routed for a recovery review

You change it to: the routing rule — this kit ships two booleans (not covered AND already paid) and a real warranty desk weighs the dollar value of the claim, the dealer's own claim history and any audit window. It is deliberately NOT the same function as coverage_verdict(), so changing WHO GETS ROUTED does not change WHAT COVERED MEANS

# Adjudicate one warranty claim record: segment, select, prompt, one model call, then a pure-code

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 3000
PLANS = {
POWERTRAIN_PARTS = ["transmission_assembly", "engine_short_block", "drive_axle"]
EMISSIONS_PARTS = ["catalytic_converter", "oxygen_sensor", "evap_canister"]
ACCESSORY_PARTS = ["infotainment_head_unit", "power_window_motor", "hvac_blower_motor"]
WEAR_PARTS = ["brake_pads", "wiper_blades", "clutch_disc"]

evals/judge.pyjudge

score field accuracy, the coverage confusion matrix, the narrative-finding grade and the recovery flag separately, and break the verdict down per deciding branch, pure code

# Score an adjudication run. PURE CODE -- gold is exact and the answer is one value per cell, so

def norm(v):
def _num(v):
def equal(field, got, want):
def deciding_branch(g):
def score(fields, records, golds):
def _matrix(rows, positive):
def score_flags(records, flags, golds):

evals/calibrate.pycalibrate

measure the completion budget the workload actually consumes before MAX_TOKENS is set — the one number in a kit that is usually inherited and never measured

# MEASURE the completion budget this workload actually needs, before setting MAX_TOKENS.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
WANT_CLASSES = ["exclusion_in_narrative", "plan_limit_beats_basic", "wear_early",
def pick(n):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2186 input and 505 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's warranty claims are entirely synthetic (tools/build_corpus.py, seed 20260822): no real manufacturer, dealer, vehicle, customer or published warranty programme exists in the corpus, and nothing was fetched from anywhere. The only outbound traffic the kit makes is one chat-completion request per claim to the configured provider, carrying the mapped sections of one claim record — the Servicing Dealer section is mapped by no field and never leaves the machine. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env, this kit's own .env (which holds only a MODEL line) or the real environment, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser. tools/shoot_ui.mjs blanks API_KEY for its free shots and refuses to start at all if something is already listening on its port, because a server it did not start may hold a real key and clicking would spend.

The experimentWe did not attack it — and three of four boundaries hold

The three boundaries that hold were confirmed by reading the code, not by a run: no code path denies a claim, opens a chargeback or contacts anybody; the routing rule and the coverage rule read only enums and numbers, so the narrative's prose cannot reach either; and the key and base URL are redacted out of any error before it is returned, with the screenshot script blanking the key outright and refusing to attach to a server it did not start. The fourth is open. An indirect prompt injection needs a field an outside party authored, and this kit has exactly one — the technician's repair narrative — which is also the field the coverage rule trusts over everything else. That combination makes it the obvious place to attack and the reason the gap is named rather than glossed. Confirmed by reading the code, not by a run, on 2026-08-22 — no attack run was fired; see redteam.why.

The first three boundaries hold, confirmed by reading the code rather than by an attack run. The fourth is open and is named as open. It matters more here than the same gap does elsewhere in this series: branch 1 of the coverage rule gives the technician's narrative authority over every structured value on the claim, so an injection that moves narrative_finding moves the verdict directly, and nothing here has tried it.

The result0 attack trials, and three of four boundaries checked here hold on evidence. The one that does not is the one this kit's own design invites: the technician's narrative is free text an outside party wrote, the coverage rule gives it authority over every structured value on the claim, and whether an instruction hidden in it could move the verdict is unmeasured.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted confusions are a REGISTER mismatch — a routine-sounding closing opinion on a claim the terms deny — and a CODED-FIELD mismatch, where the claim form's cause code contradicts what the technician described. Both measure whether a model applies the rule when the surrounding prose points the other way. Neither measures whether a model obeys an instruction hidden in the same field. Those are different failures and only two of them are measured here.

HonestyWhat this does not prove

• Whether an indirect prompt injection in the technician_narrative field could move covered on either tier. No attack run exists — and this is the field branch 1 of the rule deliberately trusts above all others.
• Whether the kit behaves safely against a hostile provider — a response body crafted to break the JSON extraction in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed claim, and nothing beyond that was tested.
• Whether a real warranty claim archive would carry anything sensitive this kit mishandles. The corpus has no personal data by construction; a real claim carries a named customer, a real VIN, a service address and a payment, and none of that path is exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_review fires when the claim is not covered AND has already been paid — covered == "no" and claim_status == "paid". Both values come out of the same reply; the rule is run afterwards in pure code, over whatever the model returned, and never over gold. A reply missing either value returns None rather than False: an unknown is not a pass.

src/extract.py::compute(), called by extract() on every claim, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by identical code. The coverage rule it reads lives in a SEPARATE function, coverage_verdict(), which is also what the corpus generator wrote gold with and what src/prompt.py states to the model — one definition, three readers, and evals/check_labels.py fails the build if the two copies of the terms drift.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a check on whether the extracted values are right. If the model misreads the odometer and then judges that misreading consistently, the claim is routed (or not routed) on a wrong number and this rule cannot tell. The self-consistency diagnostic in evals/judge.py is the closest thing to that check, it is reported separately, and it is blind to the same case.
• It is NOT any manufacturer's warranty-recovery policy. Not-covered-and-already-paid is this kit's own simplification, invented for this corpus. No published warranty programme, dealer agreement, audit procedure or chargeback rule was consulted, and none is reproduced. A real warranty desk weighs the dollar value of the claim, the dealer's own claim history and any audit or appeal window.
• It is NOT a disposition. Nothing here denies a claim, opens a chargeback, debits a dealer or credits anybody, and covered is an extracted field rather than a decision anybody is bound by.

WatchedWhat is watched, and why that one

The kit measures 18 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 11 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 55, extraction_cells 770, failures 0, refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

3 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 3 runs measured the same system — they differ on provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 3 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() or coverage_verdict() in src/extract.py, on any change to PLANS / PLAN_COMPONENTS / LABOR_OPS, on any change to data/fields.json's allowed values for covered or claim_status, and on any corpus regeneration. Changing WHO GETS ROUTED is a policy change and must be re-scored, even though it never changes what 'covered' means.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the month arithmetic is datetime, the provider is reached over urllib and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per claim — segment, select, prompt, one call, parse, compute — with no branch, no loop and no agent. Anything that looks like orchestration in a kit this size is a diagram of a straight line.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain: the JSON extraction from a fenced reply, the retry/backoff policy, the section regex, the month arithmetic and the .env reader are all code somebody has to own.
• No framework means no framework's ecosystem — no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. What ships is what is in the repository.
• The coverage terms live in TWO places — src/extract.py and tools/build_corpus.py — because a fork that keeps src/ and drops the generator must still run. That is a real duplication; evals/check_labels.py asserts the two agree before any run may spend, which converts a silent drift into a loud refusal but does not remove the second copy.
• The provider abstraction covers exactly two wire formats. A third provider is one function and one dict entry, and until somebody writes it the kit runs on two shapes.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-coverage-check on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 2 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Every claim id, vehicle id, dealership name, platform label, coverage term, component list and labor operation code is invented; no real manufacturer, model, VIN, dealer, customer or published warranty booklet is named or reproduced. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-(claim, field) exact match

did the reply return the same value gold carries, for each of the 14 fields

MethodHow the test was run

python -m evals.run --run-id <id>; evals/judge.py::score() compares every (claim, field) cell with == after light normalisation, and a numeric field within 0.005. No model is called to grade.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: gold, derived mechanically from the claim's own values.

How tight can the band be? Both published runs are at 770 of 770. There is no headroom to tolerate, so any drop at all is a real change.

Cadence: every run

The decisionWhen to reach for it

In one lineCoverage verdict confusion matrix

of every claim that really should be denied, how many did the run deny — and how many covered claims did it wrongly deny

MethodHow the test was run

evals/judge.py::score_flags() re-derives gold's verdict with the same six-branch rule the kit publishes, then builds a confusion matrix with DENIAL as the positive class. It also breaks the verdict down per deciding branch, which is the number a single accuracy figure erases.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the six-branch rule run over gold's own values.

How tight can the band be? 55 of 55 on both tiers, 1.00 recall and 1.00 precision. The free floor is at 60.0 pct, so the corpus has a 40.0-point range to detect a regression in.

Cadence: every run

The decisionWhen to reach for it

In one lineNarrative finding — the story against the coded cause

did the reply read what the technician describes, or copy what the form coded

MethodHow the test was run

evals/judge.py::score_flags() scores narrative_finding on its own, and separately over the subset of claims where the coded Cause Code disagrees with the narrated finding — the rows a reply that copies the coded field scores 0 on by construction.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the finding the generator planted, stated verbatim in the narrative it wrote.

How tight can the band be? 12 of 12 on both tiers. The floor scores 0 of 12 on it, which is what makes a 12-of-12 a real pass rather than a task nothing could fail.

Cadence: every run

The decisionWhen to reach for it

In one lineRecovery flag — the business condition

is this the claim somebody has to open a recovery on today — not covered, and already paid

MethodHow the test was run

src/extract.py::compute() runs over the reply's own covered and claim_status; evals/judge.py scores it against the SAME two-boolean rule run over gold's values.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the same two booleans run over gold.

How tight can the band be? 16 of 16 fired with 0 false alarms on both tiers. The floor is at 78.18 pct accuracy with 6 false alarms, even though it reads claim_status perfectly by regex — it inherits an opinion-derived verdict.

Cadence: every run

The decisionWhen to reach for it

In one lineSelf-consistency — no gold required

does the reply's own stated verdict survive the six-branch rule re-run over the reply's OWN extracted values, and does its stated months_in_service survive the date arithmetic re-run over its OWN two dates

MethodHow the test was run

evals/judge.py::score_flags() computes both, using no gold at all. Reported as a DIAGNOSTIC and deliberately not called this kit's guardrail — the guardrail is the business condition above.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: none — the reply is checked against itself.

How tight can the band be? On the free floor this diagnostic catches 17 of its 22 verdict errors with no labels at all — which is the honest case for running it, and also the measure of what it misses.

Cadence: every run

The decisionWhen to reach for it