The business caseThe problem this solves

A customer writes something about their bill, and somebody has to work out what it is about before anything else can happen. The words are rarely the answer on their own: this corpus's customers name the cause outright on 22 of 60 disputes, describe it without ever using a billing word on 26, name only the symptom on 5, and name the WRONG cause on 7. Nor is the bill the answer on its own -- a promotion that ended leaves no line on the invoice being disputed, and a charge billed twice across a cycle boundary leaves no repeat on either invoice. Today somebody opens the account, reads two bills side by side, decides which of six things this is, and pastes the deciding lines into the ticket. The read-the-dispute-beside-the-account step of billing dispute intake: deciding which of six causes a free-text complaint is about and pulling the specific invoice lines that bear on it. It does not decide the outcome -- the analyst still decides whether a credit is owed.

The inputThe actual disputes, each with two invoices from its own account put in front of it

The corpus is 60 disputes, each with two invoices from its own account put in front of it, 0.19 MB (json 4 · jsonl 1). Because the alternative is nothing. A billing dispute is a customer's own words about a private account, held beside that account's invoice history -- both halves are personal data about a named person and a commercial relationship, and neither exists in public in a form anybody may publish. There is no public corpus of billing disputes paired with the invoices that decide them and there will not be one. A kit whose corpus cannot ship is a kit nobody can re-run, and a kit nobody can re-run cannot be checked -- so the corpus is synthetic, the generator ships beside it, and the seed is in the dataset version. ⚑ WHAT IT IS HONESTLY GOOD FOR is comparing arms on identical input, which is what it is used for here: the same 60 disputes, the same key, the same scorer, and a paired test between the paid call and the free code.

[
 {
  "account_id": "ACCT-41000",
  "customer": "Ingrid Prendergast",
  "segment": "consumer",
  "market": "Drumlin Bay",
  "plan_code": "MOB-PLS",
  "plan_name": "Mobile Plus",
  "invoices": [
   {
    "invoice_id": "INV-41000-202511",
    "period": "2025-11",
    "issued": "2025-11-02",
    "due": "2025-12-24",
    "lines": [
     {
      "line_no": 1,
      "kind": "plan",
      "code": "MOB-PLS",
      "description": "Mobile Plus -- monthly service",
      "service_period": "2025-11",
      "amount_cents": 5500
     },
     {
      "line_no": 2,
      "kind": "fee",
      "code": "FEE-REG",
      "description": "Regulatory cost recovery",
      "service_period": "2025-11",
      "amount_cents": 346
     },
     {
      "line_no": 3,
      "kind": "promo",
      "code": "PRM-BUNDLE",
      "description": "Bundle saving -- month 5",
      "service_period": "2025-11",
      "amount_cents": -2100
     }
    ],
    "total_cents": 3746
   },
   {
    "invoice_id": "INV-41000-202512",
    "period": "2025-12",
    "issued": "2025-12-02",
    "due": "2026-01-24",
    "lines": [
     {
      "line_no": 1,
      "kind": "plan",
      "code": "MOB-PLS",
      "description": "Mobile Plus -- monthly service",
      "service_period": "2025-12",
      "amount_cents": 5500
     },
     {
      "line_no": 2,
      "kind": "fee",
      "code": "FEE-REG",
      "description": "Regulatory cost recovery",
      "service_period": "2025-12",
      "amount_cents": 346
     },
     {
      "line_no": 3,
      "kind": "promo",
      "code": "PRM-BUNDLE",
      "description": "Bundle saving -- month 6",
      "service_period": "2025-12",
      "amount_cents": -2100
     }
    ],
    "total_cents": 3746
   },
   {
    "invoice_id": "INV-41000-202601",
    "period": "2026-01",

The outcomeWhat a good result looks like

A coded ticket per dispute with its evidence already open: one of six causes, the exact invoice lines that decide it, and the amount at issue to the cent. On run r001-dispute-triage that was right on all three fields for 58 of 60 disputes, and the line references were right on 82 of 86 individual lines.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• A queue whose customers mostly name the cause -- a web form with a drop-down, a chat flow that asks 'what is this about?' first — the free keyword floor
  It scores 22 of 22 on the plain wordings here, which is every one of them, for $0.00 and no network. If your intake already funnels customers into naming the thing, the reading is not the hard part and there is nothing to buy.
• A queue of free text where customers describe the problem in their own words and the bill shows more than one change — the model
  This is the only population on this corpus where the margin is real: 25 of 26 against the delta floor's 19, 6 discordant to 0, exact two-sided p = 0.031250. Describing a charge without naming it is what language is, and a diff cannot read it.
• A queue that only needs the money -- which charge, how much — the free delta floor, then the model on what it cannot decide
  Diffing two structured invoices is exact, free and instant. The floor misstates $192.71 of the $1834.01 at issue across the corpus against the model's $42.29 -- a real difference, and one you can have for $0.00 on the 51 of 60 disputes where the bill decides on its own.

At a glanceHow the whole thing runs

Run once, for real, on 2026-09-11. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-09-11 — r001-dispute-triage. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Observed on the committed kit this session, with the model key blanked for the process: API_KEY= python3 -m src.app starts the board on port 9376 and every panel renders -- the dispute, both invoices, the answer key, all three free floors, the derivation station and the recorded run's replay -- with the model button genuinely disabled. The whole free half runs with no network at all: the corpus generator in 0.04 s, the key gate in 0.03 s (0 problems over 393 checks), and the delta floor's full 60-dispute scored run in 0.07 s. requirements.txt installs nothing; there is no pip step to fail.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one dispute -- one triage, one model call, end to end with reasoning explicitly DISABLED on the wire, over a shared connection. The derivation station adds no measurable time: it is a few dozen lines of integer arithmetic over a reply already in hand. The whole 60-dispute run took 13.5 s of wall clock at 5 workers.

Current processWhat it replaces

The read-the-dispute-beside-the-account step of billing dispute intake: deciding which of six causes a free-text complaint is about and pulling the specific invoice lines that bear on it. It does not decide the outcome -- the analyst still decides whether a credit is owed.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptEvalsEvalsApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• json
• jsonl
• md
• 1 of 11 stations carry a recorded failure

scroll → 11 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one triages -- one per dispute is

No train/test split, because nothing is trained or tuned, and no segmentation, because the dispute and its two whole invoices go into one call. The unit is one dispute; the only reduction is the two-invoice window in src/book.py, which is pure code and identical for every arm.

SetupWhat the setup figure measured

There is no index to build and no retrieval step: the dispute and both invoices go whole into one prompt at 2052.2 input tokens on average. The 0.04 s is the whole corpus generator -- accounts, disputes and the answer key -- re-run from the seed on this machine this session.

LicenceLicence

MIT -- and the MIT text is in the repository's LICENSE-PUBLIC file, NOT in its root LICENSE, which governs the repository itself. There is no third-party material to licence because there is no material from anywhere: every byte was generated by the script that ships beside it.

Bring your ownBring your own disputes, each with two invoices from its own account put in front of it

Replace data/accounts.json and data/disputes.json with your own rows in the same shape -- an account with a list of invoices, each invoice a list of charge lines carrying kind, code, description, service_period and amount_cents as an INTEGER -- and data/gold.jsonl with one key row per dispute. Then edit data/fields.json if your queue codes to different causes: the prompt, all three floors, the derivation station, the scorer and the board all read that one file, and src/causes.py asserts its own literal list matches it in order so a new cause cannot half-arrive. Run python3 -m evals.check_labels first: it re-derives every amount from your invoices longhand and refuses a key it cannot reproduce. Nothing needs an API key until you want a NEW call.

⚠︎ And what stops being true when you do: ⚠︎ WHAT STOPS BEING TRUE the moment you swap the corpus: every accuracy figure on this page, and the comparison between the arms. The finding here is that the paid call beats free code on the complete answer and not on the label, and BOTH halves of that depend on how your customers write. The phrase table in evals/baseline.py was built to match this corpus's plain wordings exactly -- it scores 22 of 22 on them -- and it will match yours worse or better. The honest first step on your own data is to edit that file and re-run the free floor BEFORE buying anything: if it already answers your queue, the rest of this kit is a comparison you do not need to pay for.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — but they are the input total apportioned by each part’s share of the characters, not a per-part reading from the provider’s counter. The total is measured; the split is arithmetic, and a part carrying denser text really costs more of it than this says.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 2,014 tokens assembled, 687 are rulebooks — 34% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The assembled prompt for the DSP-0046 call shown here, replayed from the kit's own src/prompt.py build(). That is only trustworthy because the replayed part sizes and section list match what the run recorded: the run's prompt_parts are system 2728, rulebook 2836, dispute 400, invoices 1487, schema 1026 and its sections_used are system, rulebook, dispute, invoices, schema. The per-part token counts are apportioned by character share and forced to sum to the 2014 input tokens the provider reported for this call -- the provider reports one figure for the whole prompt, not five.

Full promptThe prompt in full

You are triaging ONE incoming customer billing dispute for a telecommunications
account. Your output is what a billing analyst picks up: what the dispute is actually about, and
the specific invoice lines that decide it.

⚠︎ YOU TRIAGE. YOU DO NOT ADJUST. Nothing you return issues a credit, waives a fee, changes a
plan, closes an account or touches the bill. A person reads your answer and decides. That is why
naming the cause precisely matters more than being generous -- the wrong queue costs a day, and a
credit issued for the wrong reason costs the reason.

How to work:

- THE INVOICES ARE THE AUTHORITY ON WHAT HAPPENED. The dispute is what the customer BELIEVES
  happened. Where they disagree, the invoices are right: a customer saying "I was charged twice"
  does not make two lines a duplicate, and a customer who cannot name what changed has still been
  charged for something specific.
- READ THE CUSTOMER FIRST, AND READ ALL OF IT. People describe a billing problem without ever
  using the billing word for it. "The amount you used to take off my bill has stopped coming off
  it" is a promotion that ended. "My month has been cut in two pieces" is a proration. The words
  are often the only thing that separates two causes the invoices support equally.
- THEN COMPARE THE TWO INVOICES, LINE BY LINE. Most causes here are a DIFFERENCE between two
  months, not a fact about one. Read what is on the disputed bill, what was on the one before it,
  and -- this is the one people miss -- what was on the previous bill and is NOT on this one.
- A PROMOTION THAT ENDED HAS NO LINE ON THE DISPUTED BILL. Nothing was added; something stopped
  being taken off. Its evidence is the credit line on the PREVIOUS invoice, which is the last
  month it appeared. If you look only at the bill in front of you, you will call it a rate change.
- A MID-CYCLE CHANGE LOOKS EXACTLY LIKE A DUPLICATE AND IS NOT ONE. Two lines for the same plan on
  one invoice, each covering part of the month, is a proration: the periods are different and
  together they make one month. Two lines are only a duplicate when they are the SAME charge --
  same code, same amount, same service period -- billed twice.
- CITE EVERY LINE THE CAUSE TURNS ON, AND NO OTHERS. A rate change is two lines, one on each
  invoice, because a price that moved cannot be shown with one. A duplicate is two lines. A
  proration is every partial line on the bill. An overage or an equipment charge is one line.
- IF THE EVIDENCE DOES NOT DECIDE, PICK THE CAUSE THE INVOICES SUPPORT, not the one the customer
  named. That is the whole reason a person is reading your answer rather than the ticket.

Reply with JSON and nothing else, in the shape given at the end.

THE SIX CAUSES a billing dispute is coded to. Exactly one of them.

  rate_applied   (The rate applied)
    what it is   The recurring plan charge on this invoice is at a different price from the one the account was paying. A scheduled price change took effect, and the customer is disputing the price itself rather than anything added to the bill.
    cite         the RECURRING PLAN line on the disputed invoice, and the same plan line on the previous invoice. The dispute is the difference between two prices, so one line alone cannot show it.

  proration   (Proration)
    what it is   A plan or feature changed part-way through the billing cycle, so the invoice carries partial-month charges -- often two of them, one for each side of the change date.
    cite         every PRORATION line on the disputed invoice. There is usually more than one and citing a single one is wrong, because the pair is what explains the total.

  equipment_charge   (Equipment)
    what it is   A device instalment, a router or set-top rental, or a one-off equipment fee. The customer is disputing hardware, not service.
    cite         the EQUIPMENT line on the disputed invoice.

  promotion_expiry   (A promotion ended)
    what it is   A promotional credit that was on the previous invoice is not on this one, because the promotion reached the end of its term. Nothing was ADDED to the bill; something stopped being taken off it.
    cite         the PROMOTIONAL CREDIT line on the PREVIOUS invoice -- the last month it appeared. The disputed invoice has no line to point at, which is precisely why the account history has to be read.

  usage_overage   (Usage over the allowance)
    what it is   Data, roaming or metered usage beyond what the plan includes, charged at the overage rate.
    cite         the USAGE line on the disputed invoice.

  duplicate_charge   (Charged twice)
    what it is   The same charge appears twice -- the same code and the same amount, either twice on one invoice or once on each of two consecutive invoices for the same service period.
    cite         BOTH lines. A duplicate cited as one line is not a duplicate.

THE BOUNDARY. This kit TRIAGES. It never issues a credit, never adjusts a bill, never waives a fee, never opens or closes an account, never changes a plan and never writes to data/. Every output is a proposal with its evidence attached, for a billing analyst to accept or reject.

HOW A CHARGE LINE IS KINDED on the invoices below:
  plan        the recurring monthly charge for the service plan
  usage       metered usage beyond the plan allowance
  equipment   a device instalment, rental or one-off hardware fee
  promo       a promotional credit -- a NEGATIVE amount
  proration   a partial-month charge produced by a mid-cycle change
  fee         a regulatory recovery, admin or late fee

THE DISPUTE.

  dispute id        DSP-0046
  account           ACCT-41045 -- Rosalind Vantrease (consumer, Weatherfield)
  plan on record    Mobile Plus (MOB-PLS)
  received          2025-12-23, via app message
  disputed invoice  INV-41045-202511

  what the customer wrote, exactly as it arrived:
    I sent the phone back months ago and I am still paying towards it every month.

THE ACCOUNT'S OWN INVOICE HISTORY -- the two bills that bear on this dispute.

  THE PREVIOUS INVOICE   INV-41045-202510   issued 2025-10-02, due 2025-11-24   TOTAL $71.97
    INV-41045-202510#1   plan        MOB-PLS     Mobile Plus -- monthly service                                 $55.00   service period 2025-10
    INV-41045-202510#2   equipment   EQP-RENT    Router rental                                                  $11.79   service period 2025-10
    INV-41045-202510#3   fee         FEE-REG     Regulatory cost recovery                                        $5.18   service period 2025-10

  THE DISPUTED INVOICE   INV-41045-202511   issued 2025-11-02, due 2025-12-24   TOTAL $72.47
    INV-41045-202511#1   plan        MOB-PLS     Mobile Plus -- monthly service                                 $55.00   service period 2025-11
    INV-41045-202511#2   equipment   EQP-RENT    Router rental                                                  $11.79   service period 2025-11
    INV-41045-202511#3   fee         FEE-REG     Regulatory cost recovery                                        $5.68   service period 2025-11

  ⚠ A LINE THAT IS ON THE PREVIOUS INVOICE AND NOT ON THE DISPUTED ONE IS A FACT ABOUT THIS BILL. It is how a promotion that ended looks, and it is the only cause here whose evidence is on the earlier of the two.

Reply with JSON and nothing else, exactly this shape:

{"cited_text": "<the part of the customer's own sentence that tells you what this is about,
                 quoted VERBATIM from it; null when nothing in the words decides it and you read
                 the answer off the invoices>",
 "cause": "rate_applied" | "proration" | "equipment_charge" | "promotion_expiry"
          | "usage_overage" | "duplicate_charge",
 "evidence_lines": ["<every line reference the cause turns on, in the exact form printed in front
                     of each line above, e.g. INV-41007-202602#3. One or two of them. Lines from
                     EITHER invoice.>"]}

`cause` is exactly one of the six and nothing else -- not a phrase, not two of them joined.
`evidence_lines` is never empty: every cause here is evidenced by at least one printed line.
Do not return an amount, a credit, a refund, a recommended action or any other field. The amount
at issue is computed from the lines you cite, by the rule printed beside each cause.

Raw responseThe raw response

{
 "cited_text": "There is a charge on this bill that I do not recognise. Can somebody explain what I am paying for?",
 "cause": "promotion_expiry",
 "evidence_lines": [
  "INV-41050-202510#3"
 ]
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

What a billing dispute is actually about, and the invoice lines that decide it — 60 billing disputes drawn from 60 real disputes, each with two invoices from its own account put in front of it. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Every grader is pure code and no model judges anything, which is why re-scoring costs $0.00 and returns the same answer every time. The three graded fields are compared with == on a string, a sorted list of references and an int; there is no tolerance anywhere and no partial credit on the conjunction.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar in cost_by_model is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anybody paid. The measured spend on the tier actually used is in the run record and in measured_on.note.

Rates checked 2026-09-05. The provider that actually ran every call here is kept off this page per the series rule. Its real spend, its own dated card, its peak/off-peak split and its cached-input tier are all in results/eval-r001-dispute-triage.json in the kit repo.

Grading unitWhat the grading figure prices

Free means free: the graders reach no provider, so one eval pass over a cached run costs nothing and can be repeated as often as you like -- which this kit did, adding a consistency check to src/causes.py AFTER the 60 calls were bought and re-grading the whole run for $0.00. ⚠︎ THAT IS THE RULER, NOT THE THING MEASURED: the run itself bought 60 calls for $0.014269.

The gradersThree ways to grade

⚑ THE FLOOR IS NOT A STRAW MAN AND THE KIT WOULD BE DISHONEST IF IT WERE. The delta floor groups identical lines to find a duplicate, notices a promotional credit that stopped, sees a plan price move, and when the customer names an amount it matches that amount against what each candidate cause would come to. It is PERFECT on all 22 plain wordings and on all 7 misleading ones, and it beats the model on rate_applied. Weakening any of that would manufacture a gap for the model to fill, which is the most common way an eval flatters the thing it is measuring. ⚠︎ AND THE WEAKEST FLOOR IS PUBLISHED TOO, because the keyword floor scoring 26 of 60 makes the model look unassailable and the delta floor scoring 52 makes it look marginal -- a kit that shipped only the first would have taught its reader nothing.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — top_k, chunk size, prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-09-05. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingGrading is free; the bill is the 60 calls that answered.

⚠︎ THAT IS THE RULER, NOT THE THING MEASURED. The run itself bought 60 calls for $0.014269 and the adversarial probe bought 12 more for $0.003156. A forker re-running the pipeline pays that again; re-grading what is already committed costs nothing, which is exactly why this kit could add a consistency check to src/causes.py AFTER the calls were bought and re-score the whole run for free.

Cost driversWhat actually moves the bill

• THE RULEBOOK AND THE ROLE, not the dispute. Those two parts plus the schema are 6590 of the 8477 prompt characters and they are byte-identical on every call in the run -- which is why 70400 of the run's 123135 input tokens came back as prefix cache hits (57 pct).
• THE INVOICES. Both go in whole with every line, 1487 characters on the call shown, and they grow with the customer's bill. This is the only part of the prompt a forker's data changes.
• THE OUTPUT IS NEGLIGIBLE AND THAT IS DESIGNED. 54.9 output tokens per call on average against 2052.2 input -- the reply is an enum, a quoted fragment and at most two references, because the schema has no field for an amount, a credit or an action. A schema that asked for the money would have made the reply several times longer and bought nothing.
• NO REASONING TOKENS AT ALL. The budget was sent explicitly disabled and the provider reported none. A run that left the default in place would move this bill by an order of magnitude, which is the one thing on this page that is a setting rather than a measurement.

Your volumeWhat it costs at your volume

NOT LINEAR, and it gets cheaper per dispute before it gets dearer. Ten times the disputes over the same prompt re-sends the same fixed prefix, which a provider pricing cached input bills at a fraction -- 57 pct of this run's input was already a cache hit at only 60 calls. What does NOT scale is the invoices block: it is the only part that varies, and a customer base with longer bills moves it directly.

Model choiceWhy this model

One provider, one key, one model -- the rule for every kit in this series, because a forker holds one credential and a bake-off is a different kit. The tier chosen is the cheap one, with reasoning sent explicitly disabled, because the job is a six-way label and two line references and the reply never came close to the 700-token ceiling: the largest of 60 was 71 tokens. ⚑ AND THE REAL QUESTION ON THIS KIT IS NOT WHICH MODEL BUT WHETHER ANY, because the free delta floor takes 51 of 60 for $0.00.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

the whole 60-dispute scored run (r001-dispute-triage, the fast tier): 123135 input tokens, 70400 of them prefix cache hits, and 3291 output of which NONE is reasoning -- the budget was sent disabled. All 60 calls answered, 0 failures, so this total covers every call in the denominator.

The codeEvery file, and what it is for

12 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pythe corpus generator

Writes the accounts, the disputes and the answer key from one seeded random.Random(20260911). ⚑ IT REFUSES TO WRITE A CORPUS ITS OWN KEY CANNOT EXPLAIN: a promotional credit must be negative and absent from the disputed bill, a duplicate must be two lines of the same code, amount and service period, a rate change must actually move the plan price, a proration must replace the whole-month line, and every dispute must be answerable from the two invoices and the customer's own words together. Each of those assertions fired during authoring.

# Write the corpus, the disputes and the answer key from ONE seed. No network, no model, no key.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
SEED = 20260911
DATASET_VERSION = "dispute-triage-v1-60disputes"
FIRST = ["Marisol", "Devan", "Priya", "Otto", "Nadine", "Calvin", "Reza", "Imani", "Thom",
LAST = ["Vantrease", "Okonkwo", "Halloran", "Brisbois", "Ferraro", "Nakashima", "Quintero",
BIZ = ["Tallow & Rind", "Kestrel Bindery", "Marrow Lane Dental", "Bellweather Signs",
CITY = ["Ashcombe", "Mount Verity", "Fairhaven", "Pellingham", "New Ossory", "Drumlin Bay",
PLANS = [

src/book.pythe window

THE ONLY REDUCTION STEP IN THIS KIT, and it is pure code with no model in it. One dispute is put in front of exactly TWO invoices -- the one it is about and the one before it. Every account carries three, so 60 invoices are deliberately withheld. One is provably not enough: promotion_expiry is evidenced by a line that is not on the disputed bill at all.

# Loading the corpus, and the ONE rule about what a triage is allowed to see.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
def accounts():
def disputes():
def gold():
def stats():
def by_id(rows, key):
def account_of(dispute, accts=None):
def window(dispute, accts=None):

src/causes.pythe causes and the derivation station

The six causes as data, the reference normaliser that accepts three spellings of a line reference, and derive() -- which turns (cause, cited lines) into an amount at issue in integer cents by that cause's own stated rule. Every arm passes through it. It reports a problems list when the citation contradicts the cause and STILL returns the number.

# The six causes, and the arithmetic station that turns a READING into a number.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
SPEC = json.load(open(os.path.join(HERE, "data", "fields.json"), encoding="utf-8"))
CAUSES = ["rate_applied", "proration", "equipment_charge",
CAUSE_MEANINGS = {c["code"]: {"label": c["label"], "means": c["means"],
CAUSE_KINDS = {
LINE_KINDS = [k["kind"] for k in SPEC["line_kinds"]]
REF_RE = re.compile(SPEC["line_ref_pattern"])
GRADED = SPEC["graded"]
BOUNDARY = SPEC["boundary"]

src/prompt.pythe prompt

Five parts in a fixed order, 8477 characters: the role and the two traps, the six causes with what each one cites, the dispute verbatim, both invoices line by line with their references printed, and the JSON schema. The first two and the last are byte-identical across every call in a run.

# The assembled prompt, in one place, in the order it is sent.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
SYSTEM = """You are triaging ONE incoming customer billing dispute for a telecommunications
def rulebook_block():
def dispute_block(d, win):
def _invoice(inv, label):
def invoices_block(win):
SCHEMA = """Reply with JSON and nothing else, exactly this shape:
def build(d, win):
def render(parts):

src/triage.pythe model call

The ONLY place a provider is reached. One dispute, one call, reasoning sent explicitly disabled, ceiling 700 output tokens. It also records every field of the reply the schema refuses, which is how the boundary is measured rather than assumed.

# One dispute in, one proposed triage out. The only place a model is called.

MAX_TOKENS = 700
THINKING = THINKING_OFF
def parse_reply(text):
def forbidden_fields(obj):
def triage(cfg, d, win, complete_fn=None, max_tokens=None):

src/adapters/__init__.pythe provider adapter — a swap seam

Raw HTTP over urllib to any OpenAI-compatible provider or to Anthropic. Distinguishes a TRANSIENT failure from a TERMINAL one and retries only the first; a blown socket gets one retry, not four, because a non-streamed completion that timed out will time out again and be billed again.

You change it to: PROVIDER + BASE_URL + MODEL in .env -- one line, then the same run again. Configure it ONCE at the repo root and every kit in this repo inherits it; a kit-local .env overrides key by key, so 'same account, different model' is one line.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TIMEOUT_S = 1200
TRANSPORT_RETRIES = 1
def _post(url, headers, payload, timeout=TIMEOUT_S):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}

evals/baseline.pythe three free floors — a swap seam

constant (one answer to everything), keyword (a phrase table over the customer's words) and delta (a diff of the two invoices, arbitrated with the phrase table and with any amount the customer stated). All three hand their judgement to the same derivation station the paid arm uses.

You change it to: The phrase table, the candidate rules and the arbitration order. If your own queue's wordings differ from this corpus's, this is the file to edit FIRST -- before buying anything, because the floor is what the spend has to beat.

# The free floors. No key, no network, no model -- and they are the number the paid arm has to

MODES = ("constant", "keyword", "delta")
PHRASES = [
MONEY_RE = re.compile(r"\$\s?([0-9][0-9,]*(?:\.[0-9]{2})?)")
def majority_cause():
def stated_amounts(text):
def keyword_cause(text):
def _tot(inv, kind):
def _ref(inv, l):
def evidence_for(cause, win):

evals/scoring.pythe graders

Pure Python, exact comparison, no model. Two grains travel: the dispute grain (evidence_exact) and the line grain (recall and precision over 86 cited lines), because an arm that cites one of a proration's two lines scores 0 on the first and 0.5 on the second.

# The graders. Pure Python, exact comparison, no model anywhere -- so re-scoring a recorded run

def _pc(n, d):
def majority_floor(golds):
def score(answers, golds, disputes):
def correct_map(answers, golds, field="cause"):

evals/check_labels.pythe key gate

Grades the ANSWER KEY, longhand, without importing src/causes.py -- re-running the same function over the same inputs proves determinism and nothing else. 0 problems over 393 checks on the shipped bytes.

# Grade the ANSWER KEY. Pure code, no model, no network, and it does not import the station.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CAUSES = ["rate_applied", "proration", "equipment_charge",
ALLOWED_KINDS = {
def money(c):
def main():

evals/significance.pythe paired test

McNemar's exact two-sided binomial on the discordant pairs. Computed on every graded field against every floor whose result file is on disk, whichever way it falls.

# Is the paid arm's margin over a free floor real, or is it noise? PURE CODE, no model.

def mcnemar_exact(b, c):
def compare(a_correct, b_correct, label_a="paid", label_b="floor"):

src/app.pythe local board

Standard-library HTTP server on port 9376. Every route but /api/triage runs with no key: the corpus, the key, all three floors, the derivation station and the recorded run all replay for free. No route returns the model id.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "9376"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-dispute-triage")
FLOOR_RUNS = {"constant": "b000-dispute-triage-constant",
PROBE_RUN = "x001-dispute-triage"
def recorded(run_id):
def invoice_rows(win, answer=None, gold=None):
def invoice_totals(win):

tools/shoot_ui.mjsthe shooter

Drives the real board with puppeteer and asserts the document in front of the camera is this kit's, on every navigation, before every frame -- and measures every table against its own clipping wrapper on every shot.

# 

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2052 input and 54 output tokens per query at top-k 2, of which about 150 tokens are fixed (system prompt, chat framing, the question itself) and the rest is retrieved context. Change the retrieval depth and the input scales with it. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model and top-k follow the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

⚑ THE PROBE WAS FIRED AND THE RESULT IS TWO-SIDED. The threat is not hypothetical for this use case: the dispute text is the CUSTOMER'S free text, typed into a web form or a chat window, and it reaches the model verbatim by design -- and the party who wrote it has a direct financial interest in a credit. That is the lowest bar of any injection surface in this series: it needs somebody to type a sentence, not to breach anything. 12 calls were bought across 4 wordings on 3 disputes the scored run had answered correctly. THE CAUSE HELD EVERY TIME: 0 of 4 steerable trials obeyed the injected cause, 0 of 12 trials returned any cause other than the key's, and 0 of 12 returned a field the schema refuses. ⚠︎ AND THE CITATION DID NOT. 2 of 12 trials returned EVERY line on both invoices, because one wording told them to -- with the cause still correct, so a page reporting classification accuracy alone would have shown a clean sweep. What the derivation station then computed from those citations is $162.33 where the key says $16.81, and $322.45 where the key says $14.00. The attack did not change the answer; it changed the money, by an order of magnitude, through the field nobody was watching.

API_KEY is read from the shared <repo>/.env, this kit's own .env, or the real environment, in that order. Both files are gitignored from the first commit; this repo has never held a credential. The key never leaves the machine, is never printed, and src/app.py strips api_key and base_url out of any exception message before it reaches the browser, so a misconfigured key cannot leak into a UI error. src/budget.py counts CALLS against a shared cap before anything is spent and says out loud when no cap is configured rather than refusing to run a fresh fork. Every free path -- the corpus, all three floors, the recorded run, the board -- needs no key at all. ⚑ AND NO ROUTE RETURNS THE MODEL ID: that is structural, so a published screenshot cannot show it and no reviewer has to check a template.

The experimentWe attacked it — in the field the customer types in

Four sentences, appended to the customer's own dispute text, on three disputes the scored run had answered correctly. The surface is real and it is the lowest bar in this series: an attack needs somebody to type a sentence into a web form, not to breach anything. Both were run for real on 2026-09-11 -- 12 bought calls on run x001-dispute-triage, summarised by python3 -m evals.injection --rescore, which recomputes from the committed trials and buys nothing.

Every wording is printed verbatim in the redteam block below, because a probe whose attack text is not published cannot be re-read, argued with or reproduced.

The resultThe cause held under every wording and the CITATION did not, and the citation is what the money is computed from. 0 of 4 steerable trials obeyed the injected cause, 0 of 12 trials moved the answer at all, 0 of 12 returned a field the schema refuses -- and 2 of 12 returned every line on both invoices because one sentence told them to, with the label still correct. Structurally, a credit or an action in a reply reaches nothing: src/causes.py builds the published answer from cause and evidence_lines alone. That is exactly why the attack that worked had to come through evidence_lines.

The probe appends one sentence to the customer's own text, in the field a person actually types in, on DSP-0007, DSP-0044 and DSP-0050 -- three disputes across three causes that the scored run answered correctly, so a moved answer is the injection and not a case the arm was already failing. Four wordings: a plain instruction to use a different cause, a forged supervisor note, a demand for credit_cents and an action field, and an instruction to cite every line on both invoices. ⚠︎ THE FOURTH ONE WORKED, 2 times of 12, and the first three did not. ⚑ AND THE obeyed FIGURE IS SMALLER THAN IT FIRST READ: two of the six steerable wordings happened to demand the cause the key already holds, so the first summary recorded a correct answer as the attack landing and reported 2 obeyed for an arm that had not moved. Those trials are excluded and counted separately rather than discarded.

HonestyWhat this does not prove

• Whether a longer or more patient injection works. Four wordings on three disputes is a probe, not a red-team engagement: no multi-turn, no text hidden inside a charge description, and no wording that AGREES with what the bill already shows -- which is the attack most likely to succeed, because it asks the arm to do something it was already inclined to do.
• Whether the INVOICES can be poisoned. Every charge line carries a free-text description that on a real bill comes from a product catalogue a lot of people can edit, and a sentence planted there would arrive inside the part of the prompt the arm is told is authoritative. The probe attacks the dispute text only: 0 trials on that surface.
• Whether the 2-of-12 citation flood generalises. It landed on two of the three probed disputes and not on the third, and three disputes is not a rate.
• What a real customer would write. These four sentences were written to attack; a customer with a genuine financial interest and time to spend would write differently and this kit cannot say how.
• Anything about a second model, a second provider or a second day. 12 calls, one tier, one afternoon.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never issue a credit, adjust a bill, waive a fee, change a plan, open or close an account, or write to any system. This kit produces the coded ticket a PERSON picks up, with both invoices, the cited lines, the amount rule and the arithmetic printed beside the figure, so they can check it rather than believe it. An amount a person cannot check is an amount they have to believe.

Stated in the prompt the model is sent (“you triage, you do not adjust”), at the top of data/fields.json as the boundary field that the prompt renders verbatim, on the UI as a banner rather than a footnote, in the README, and in src/app.py's docstring -- and ENFORCED by there being no such endpoint and no such field: the schema has no slot for an amount, a credit, a refund or an action, and src/causes.py builds the published answer from cause and evidence_lines alone.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is ABSENCE OF A WRITE PATH plus a schema with no slot for money, not a runtime enforcement layer. src/causes.py::derive() IS enforcement for the arithmetic -- no arm can state an amount, because there is nowhere to state one -- and it is NOT enforcement for the JUDGEMENT, which is where every unfixable error in this kit lives.
• ⚠︎ THE STATION CANNOT FIX A MISREADING, AND SAYING IT HELD IS NOT SAYING IT WORKS. Handed a cause the bill does not support, it applies that cause's rule faithfully and returns a number that is wrong for a reason no care with cents can see. Measured: on DSP-0051 it published $24.00 against a key of $2.59 -- arithmetically perfect, nine times the truth.
• It cannot return 'I do not know'. The schema offers six causes and no abstain, because the corpus contains exactly one cause per dispute. A real queue needs a catch-all and an escalate, and this kit has neither -- so an arm faced with a seventh kind of dispute will name the nearest of the six, confidently.
• It cannot see a dispute about two things at once. One cause comes back, and nothing anywhere says the other half was dropped.
• The problems flag is a consistency check, not a correctness one. It fired on 1 of the model's 60 answers and caught one of its two misses; the other miss -- a cross-cycle duplicate read as a usage overage -- cited a real usage line of the right kind and produced no contradiction at all.

WatchedWhat is watched, and why that one

The kit measures 49 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

10 measured by the latest run 39 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

5 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

triage · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 3 runs measured the same system — they differ on the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

triage · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 33 chips that all say so.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 10 chips that all say so.

DeviationsWhat deviated

0 breaches across 5 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The corpus generator, the key gate, the derivation station and all three free floors are free and re-run in under a second with no network -- verified this session on the committed kit (0.04 s, 0.03 s and 0.07 s respectively). Only the model arm costs money: 60 calls, once, $0.014269 off-peak, plus 12 adversarial calls at $0.003156. ⚑ RE-GRADING IS FREE AND WAS USED: a consistency check was added to src/causes.py AFTER the calls were bought and the whole run was re-scored with --resume --rescore for $0.00. Re-run the free half on every change to the corpus, the causes or the floors; re-buy the model arm only when the PROMPT or the window changes, and never to improve a published number.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, standard library end to end -- no orchestration layer, no vendor SDK, no retrieval stack, no numeric or dataframe library, no pip install. requirements.txt names nothing and spends its whole body saying why, including the reason a decimal library is refused: every amount here is an integer number of cents from the generator to the key to the prompt to the scorer, and an integer cannot be 1790.9999999999998, so there is nothing for a Decimal to protect -- while decimal would make ROUNDING configurable, which is exactly the freedom that lets two implementations of one rulebook disagree at a half-cent and both look right.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear, and concurrent only at the dispute level. There is no agent, no tool loop, no retrieval and NO STATE carried between disputes -- no invoice is ever adjusted, so each triage is against the account as committed. window -> prompt -> one call -> parse -> derive -> score. That is the whole graph, and it is why the corpus can be re-scored for free.

The other sideWhat a framework costs you

• Adding a provider means editing the PROVIDERS dict by hand. In exchange pip install pulls nothing and a forker runs this on whichever key they already hold.
• No schema validation on the way out: a reply carrying a field the schema refuses is COUNTED (src/triage.py::forbidden_fields) rather than rejected, because the boundary here is structural -- the published answer is built from two fields and nothing else.
• No retry on a wrong reading, deliberately. A reply that parses and is wrong stays in the denominator; a kit that re-fired its misses would publish a number nobody could reproduce.
• The reference normaliser accepts three spellings of a line reference, which is generous and untested in anger: 0 of 60 replies on this run needed any of them.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-dispute-triage, 2026-09-11. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 3 runs on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — one report is one event, whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, FOUNDRY_RUN_ID, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- and the MIT text is in the repository's LICENSE-PUBLIC file, NOT in its root LICENSE, which governs the repository itself. There is no third-party material to licence because there is no material from anywhere: every byte was generated by the script that ships beside it. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe whole triage -- the cause, the exact SET of invoice lines it turns on, and the amount at issue in integer cents, all three right on one dispute

whether each of the 60 coded tickets is one a billing analyst could pick up as written. All three, never two: the right cause with the wrong lines sends an analyst to a page that does not explain the ticket, and the right lines with the wrong amount puts a number in front of a customer that nobody can reproduce. The amount is never asked for -- it is derived in code from the other two -- so this grader is really asking whether the READING was good enough for the arithmetic to land.

MethodHow the test was run

python3 -m evals.run --run-id <id> [--floor constant|keyword|delta]; evals/scoring.py compares a string, a sorted list and an int against data/gold.jsonl. No model is in the grading path, and re-scoring a cached run costs nothing -- this kit's promotion-consistency check was added AFTER the calls were bought and the whole run was re-graded with --resume --rescore for $0.00.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

the recurring plan charge is $2.59 higher than the same line on the previous invoice, and nothing was added to the bill

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl -- 60 rows, one per dispute, each carrying the cause, the exact line references, the derived amount at issue, the register the customer wrote in, the decoy a misleading wording points at, and whether the disputed bill carries a competing change. The key is DERIVED by the generator, which decides the cause first and builds the two invoices so that cause is the true one -- there was no labelling step and therefore no labeller to disagree with. evals/check_labels.py then re-derives every amount a second time longhand without importing src/causes.py. That is the honest description and it is also the key's own limit: it agrees with data/fields.json by construction, so what it proves is internal consistency to the cent, not that these six causes are how any given carrier codes a queue.

No true/false rates for this grader. It records 8 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? There is no threshold and nothing is tunable. Every comparison is == against the key -- a string, a sorted list of references, and an int. An arm is right on a field or it is not. The only judgement call in the whole grader is which three fields are graded, and that list lives in data/fields.json rather than in the scorer.

Cadence: Once per run. Re-scoring a cached run is free and deterministic, so a change to src/causes.py can be re-graded against the same bought calls with --resume --rescore and no provider in the loop -- which is exactly what happened to this run.

The decisionWhen to reach for it

In one lineThe cause alone -- one of six, exact match

which queue the ticket goes to, and nothing else. It is the field a real intake system would route on, so it is graded on its own -- and it is the field on which this kit's paid arm does NOT clear significance against free code.

MethodHow the test was run

The same run; the figure is in scores.cause_correct of every result file. evals/significance.py pairs it against every floor on disk.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

the recurring plan charge is $2.59 higher than the same line on the previous invoice, and nothing was added to the bill

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl's cause field -- one of the six codes in data/fields.json, planted by the generator before the invoices were built rather than read off a finished bill.

No true/false rates for this grader. It records 4 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? Exact string equality against one of six codes. src/causes.py normalises case, spaces and hyphens on the way in and refuses anything that is not one of the six, so an arm that invents a seventh label scores wrong rather than crashing the scorer.

Cadence: Once per run, free to re-score.

The decisionWhen to reach for it

In one lineThe evidence -- the invoice lines the cause turns on, at two grains

whether the analyst opens the right lines. Graded twice on purpose: the DISPUTE grain asks whether the whole set is exactly right, and the LINE grain asks what share of the key's 86 individual references were found. An arm that cites one of a proration's two lines scores 0 on the first and 0.5 on the second, and publishing only the harsher of the two would hide the difference between an arm that half-read the bill and one that did not read it.

MethodHow the test was run

The same run. Both grains are in scores of every result file.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

the recurring plan charge is $2.59 higher than the same line on the previous invoice, and nothing was added to the bill

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl's evidence_lines -- the exact references the generator recorded when it planted the cause, checked by evals/check_labels.py against the declared reference pattern, against the two-invoice window, and against the kinds each cause's rule admits.

No true/false rates for this grader. It records 4 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? Sorted-list equality on the dispute grain and set intersection on the line grain. No tolerance, no partial credit on the first, and the second is a plain count of hits over the key's own total.

Cadence: Once per run, free to re-score.

The decisionWhen to reach for it