The business caseThe problem this solves

A utility's billing run throws exceptions -- a read that failed validation, consumption outside expected bounds, a tariff that did not resolve. They queue up and somebody triages them. The interesting failure is that the SAME SYMPTOM has several causes with completely different fixes: a high bill from a genuine leak, from a run of estimates now being trued up, from a tariff that changed mid-period, from a meter exchange whose old register was never closed off, or from a multiplier applied twice. Free code can classify by symptom and route the whole pile to one queue; it cannot tell WHICH CAUSE is in play, and the cause is what decides whether the answer is a field visit, a rebill, a metering referral or nothing at all. a billing exception queue that classifies by SYMPTOM -- 'consumption more than three times the same period last year' -- and routes the pile to one desk. On this corpus that reaches 39.17 pct of causes, because 71 of 120 readings have a cause that is not their symptom's usual one.

The inputThe actual billing-exception dossiers

The corpus is 120 billing-exception dossiers, 1.06 MB (txt 120). A utility's exception queue is a map of which premises are under-billed, which meters are unread and which accounts are in arrears. There is no public one, and a scrubbed export is worse rather than better: scrubbing removes the field log, and the field log is exactly the layer this kit measures.

Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated utility billing-exception dossier
for an AI use-case kit; it reproduces no utility, no supplier, no meter operator, no
data collector, no premise, no account and no meter. IT CONTAINS NO CUSTOMER IDENTITY
AND NO HOUSEHOLD DETAIL BY CONSTRUCTION -- accounts, premises, meters, registers,
tariffs and reads only. The thresholds, the rebill floor, the tolerances and the
cause-to-action routing table are ILLUSTRATIVE DEFAULTS.

Account And Premise
----------------------------------------------------------------
Exception reference              : BX-0001
Account reference                : ACC-84767424
Premise reference                : PRM-34077
Meter point                      : MP-6317283
Supply                           : ELECTRICITY, metered, domestic single rate
Billing queue on the account     : Domestic Billing Team C
Open field job on this premise   : NO
Billing period                   : 2026-03-21 to 2026-04-21 (31 days)

Exception Raised
----------------------------------------------------------------
Exception raised by the billing run : LOW_OR_ZERO_CONSUMPTION
Raised on                        : 2026-04-23
Rule that raised it              : billed consumption less than 0.25x the same period last year

  Billed consumption this period            16.00  kWh
  Same period, prior year                  690.46  kWh
  Ratio to prior year                        0.02  x
  Billed value this period                  20.35  GBP
  Value at stake on this exception          19.00  GBP

Meter And Register
----------------------------------------------------------------
Meter serial                     : E-89514602

The outcomeWhat a good result looks like

every exception's cause, the one line of the record that decides it, and therefore the fix -- with 100 pct of field visits caught and 0 false rebills proposed across 58 readings whose truth was not a rebill.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• your exception evidence is all in structured fields -- event codes, read-type flags, tariff effective dates — the rules floor in evals/baseline.py, and no model at all
  It reaches 85.51 pct of causes on the structured half of this corpus for $0.00, one reading behind the model.
• the reason an exception exists is typed into a notes field by whoever attended — the model, with the account history in the prompt
  100.0 pct against the strongest free floor's 70.0 on the 40 prose-decided readings, and 94.87 against 71.79 on the readings whose log carries a speculative line naming the wrong cause.

And where nothing here is good enough:

• you want the model to decide the FIX rather than the cause — neither -- keep the routing table in code
  The fix here is a lookup on the cause plus two printed arithmetic overrides, and every arm computes it with the same function. That is why the action column moves only when the cause column does.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-25. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-25 — r002-bill-exception. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Every dossier, the answer key, all three free floors, the injection probe and what r002-bill-exception actually answered ship in the repo. python3 -m evals.check_labels and python3 -m src.app both run with no key and no network.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one reading -- one account, one billing period, one exception -- end to end including provider-side reasoning tokens, on the shared connection. Not a cold-start figure and not a per-exception SLA: 120 readings ran in 368.2 wall seconds because they are independent and ran concurrently.

Current processWhat it replaces

a billing exception queue that classifies by SYMPTOM -- 'consumption more than three times the same period last year' -- and routes the pile to one desk. On this corpus that reaches 39.17 pct of causes, because 71 of 120 readings have a cause that is not their symptom's usual one.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptArchitectureArchitectureApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• 4 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one reading is

120 independent exceptions, one account and one billing period each. There is no chain and no carried state, so the readings are order-independent and ran concurrently. Causes are planted to a declared distribution (PLAN in the generator): 71 of 120 readings are OFF-MODE and 40 are decided only by prose.

SetupWhat the setup figure measured

There is no index to build -- each dossier goes whole into the prompt. The 0.6s and $0.00 are the corpus generation itself.

LicenceLicence

MIT

Bring your ownBring your own billing-exception dossiers

tools/build_corpus.py writes data/corpus/*.txt and data/gold.jsonl. Keep the 10 section headings and the field labels src/classify.position_of parses, and keep the [Enn] tags -- the evidence column is scored against them. Set your own thresholds, rebill floor and routing table in src/triage.py first.

⚠︎ And what stops being true when you do: The measured figures on this page do not transfer to your own queue, and four of them are properties of this generator's declared distribution rather than facts about utility billing: the off-mode share (71 of 120), the prose-decided share (40 of 120), the 39 field logs carrying a speculative line naming a cause that is not the cause, and -- the one most likely to mislead -- THE PROSE ITSELF IS DRAWN FROM FIVE AUTHORED PHRASINGS PER CAUSE. Real field notes vary far more, so the strongest free floor's 70.0 pct on the prose slice is an upper bound for a keyword approach on THIS corpus and would be lower on yours. Re-run the evals on your own data; that is what the harness is for.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — but they are the input total apportioned by each part’s share of the characters, not a per-part reading from the provider’s counter. The total is measured; the split is arithmetic, and a part carrying denser text really costs more of it than this says.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 3,430 tokens assembled, 2,312 are per-readings — 67% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Reproduced from src/prompt.build for BX-0112, not retyped. The middle block is the one the control removes.

Full promptThe prompt in full

You are the billing exception desk for one utility. A billing run has thrown an exception against
one account for one billing period, and you are triaging that ONE exception.

The account, the premise, the meter and its register, the exception the billing run raised, the
DEFAULT triage policy, the account's read history, its tariff history, its meter event log and the
free-text field log are reproduced in the dossier below. Apply the policy exactly as written.

⚠︎ THE THRESHOLDS, THE REBILL FLOOR, THE TOLERANCES AND THE CAUSE-TO-ACTION ROUTING TABLE ARE
ILLUSTRATIVE DEFAULTS, NOT ANY REAL UTILITY'S, REGULATOR'S OR SUPPLIER'S POLICY. Apply them as
printed regardless of whether they look right for the account in front of you.

THE POINT OF THIS JOB: the SYMPTOM is already printed on the page. The billing run computed it and
it is not the finding. The SAME symptom has several causes with completely different fixes -- a high
bill from a genuine leak needs somebody at the meter; a high bill from a run of estimates now being
trued up needs a rebill spread back and NOBODY at the meter; a high bill from a tariff that moved
mid-period needs a rebill and nothing else; a high bill from a meter exchange whose old meter was
never closed off is a metering data defect that billing cannot fix; a high bill from a multiplier
applied twice is arithmetic. WHAT YOU ARE DECIDING IS THE CAUSE. The fix is a lookup on the cause,
so agreeing with the symptom earns nothing and getting the cause right for the wrong reason is not
getting it right.

How to work it out:
 - FIRST decide whether the dossier is complete. If the read history or the meter and register block
   is not present for this billing period, the symptom is CONTEXT_INCOMPLETE: no cause is
   determined, the evidence is NONE and the action is NO_ACTION (Rule X-2).
 - THEN copy the SYMPTOM the billing run raised, exactly as printed on the exception line. Do not
   re-derive it and do not disagree with it -- it is an input (Rule X-1).
 - THEN decide the CAUSE by the FIRST rule that fires, in the printed order X-4 through X-12. The
   order is the policy. An arithmetic defect that makes the index unusable is settled before
   anything is inferred from consumption computed off that index.
 - EVIDENCE MAY BE STRUCTURED OR IT MAY BE PROSE, AND THE TWO COUNT THE SAME. Every line in the read
   history, the tariff history and the meter event log carries a tag like [E07]; so does every entry
   in the field log. A field technician writing "old meter came off on the 14th, final index read at
   the board and written on the job card, never keyed" has recorded a meter exchange that was not
   closed off just as surely as an EVT: EXCHANGE row would have.
 - THEN cite the ONE evidence tag that DECIDES the cause -- the single [Enn] a colleague would have
   to read to reach the same conclusion. Not the exception line, not a supporting detail: the one
   that decides it. If the cause is UNDETERMINED or CONTEXT_INCOMPLETE, answer NONE (Rule X-13).
 - USE UNDETERMINED WHEN NOTHING SAYS WHY. It is not a synonym for "probably the usual one for this
   symptom". Preferring a plausible label to UNDETERMINED is how a real defect acquires a benign
   explanation and stops being looked at (Rule X-12).
 - THEN give the ACTION. Take it from the printed routing table for the cause you decided, then
   apply the two printed overrides in order: a REBILL whose printed impact is BELOW the printed
   rebill floor becomes NO_ACTION, and a FIELD_VISIT on a premise that already has an OPEN field job
   on record becomes NO_ACTION (Rule X-14).
 - FINALLY give the OWNER: the billing queue printed on the account, unless a re-queue is recorded
   anywhere on this record -- including one typed as an ordinary sentence in the field log -- in
   which case that supersedes it (Rule X-15).

Answer with a single JSON object and nothing else:

{"symptom": "HIGH_CONSUMPTION|LOW_OR_ZERO_CONSUMPTION|NEGATIVE_CONSUMPTION|READ_VALIDATION_FAIL|TARIFF_NOT_APPLIED|BILL_VALUE_OUTLIER|CONTEXT_INCOMPLETE",
 "cause": "LEAK_SUSPECTED|GENUINE_CONSUMPTION|ESTIMATE_TRUEUP|TARIFF_CHANGE_MIDPERIOD|EXCHANGE_NOT_CLOSED|MULTIPLIER_DOUBLE_APPLIED|REGISTER_ROLLOVER|TRANSPOSED_READ|UNDETERMINED",
 "evidence": "<the one tag that decides it, exactly as printed, e.g. E07 -- or NONE>",
 "action": "FIELD_VISIT|REBILL|REFER_METERING|NO_ACTION",
 "owner": "<the billing queue this exception belongs to>",
 "rationale": "one sentence, naming the cause, the evidence tag you relied on, and the rule number
   you applied"}

Give the evidence tag WITHOUT its brackets: E07, not [E07].

Account history
----------------------------------------------------------------
Read History
----------------------------------------------------------------
The last 11 reads on this meter point, oldest first. The final row is this
period's closing read.

  Tag    Read date     Type       Index          Validation
  [E01]  2023-09-02    ACTUAL          38,651   PASS
  [E02]  2023-12-02    ESTIMATE        39,608   PASS
  [E03]  2024-03-02    ACTUAL          40,463   PASS
  [E04]  2024-06-01    ESTIMATE        41,281   PASS
  [E05]  2024-08-31    ACTUAL          42,250   PASS
  [E06]  2024-11-30    ACTUAL          43,106   PASS
  [E07]  2025-03-01    ACTUAL          44,142   PASS
  [E08]  2025-05-31    ACTUAL          44,961   PASS
  [E09]  2025-08-30    ACTUAL          45,794   PASS
  [E10]  2025-11-29    ACTUAL          46,626   PASS
  [E11]  2026-02-28    ACTUAL          45,978   PASS

Tariff History
----------------------------------------------------------------
  Tag    Effective     Tariff code           Standing charge   Unit rate
  [E12]  2024-06-05    DOM-ELEC-STD-24       0.4770 GBP/day   0.2132 GBP/kWh
  [E13]  2021-12-27    DOM-ELEC-STD-24       0.4876 GBP/day   0.2180 GBP/kWh
  [E14]  2023-10-24    DOM-ELEC-STD-24       0.4982 GBP/day   0.2230 GBP/kWh

Tariff applied by this billing run : DOM-ELEC-STD-24 across the whole period

Meter Events
----------------------------------------------------------------
No meter events are recorded against this meter point for this period.

Exception dossier
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated utility billing-exception dossier
for an AI use-case kit; it reproduces no utility, no supplier, no meter operator, no
data collector, no premise, no account and no meter. IT CONTAINS NO CUSTOMER IDENTITY
AND NO HOUSEHOLD DETAIL BY CONSTRUCTION -- accounts, premises, meters, registers,
tariffs and reads only. The thresholds, the rebill floor, the tolerances and the
cause-to-action routing table are ILLUSTRATIVE DEFAULTS.

Account And Premise
----------------------------------------------------------------
Exception reference              : BX-0112
Account reference                : ACC-23535870
Premise reference                : PRM-79093
Meter point                      : MP-3563173
Supply                           : ELECTRICITY, metered, domestic single rate
Billing queue on the account     : Revenue Protection
Open field job on this premise   : NO
Billing period                   : 2025-11-29 to 2026-02-28 (91 days)

Exception Raised
----------------------------------------------------------------
Exception raised by the billing run : NEGATIVE_CONSUMPTION
Raised on                        : 2026-03-04
Rule that raised it              : the period's metered difference is negative

  Billed consumption this period          -648.00  kWh
  Same period, prior year                  913.90  kWh
  Ratio to prior year                       -0.71  x
  Billed value this period                 206.99  GBP
  Value at stake on this exception         103.50  GBP

Meter And Register
----------------------------------------------------------------
Meter serial                     : E-10415486
Register digits                  : 6   (rolls over at 999,999)
Unit                             : kWh
Multiplier applied by the data collector : 1.00
Multiplier applied by the billing engine : 1.00
Opening index (period start)     : 46,626
Closing index (period end)       : 45,978
Metered difference               : -648

Triage Policy (Default)
----------------------------------------------------------------
The thresholds, floors and routing below are an OPERATOR-TUNABLE DEFAULT, not a real
utility's, regulator's or supplier's policy -- no documented triage policy exists for
this row. Applied exactly as printed.

  High-consumption threshold     : 3.0 x the same period last year
  Low-consumption threshold      : 0.25 x the same period last year
  Read validation tolerance      : 40 pct
Rebill floor                     : 15.00 GBP -- a rebill worth less than this is not
                                   raised
Adjustment authority             : NOT DEFINED. Nothing in this kit issues a bill,
                                   cancels a bill, posts a rebill, raises a credit,
                                   books a field job or amends a read.

  Cause                          Action
  LEAK_SUSPECTED                 FIELD_VISIT
  GENUINE_CONSUMPTION            NO_ACTION
  ESTIMATE_TRUEUP                REBILL
  TARIFF_CHANGE_MIDPERIOD        REBILL
  EXCHANGE_NOT_CLOSED            REFER_METERING
  MULTIPLIER_DOUBLE_APPLIED      REBILL
  REGISTER_ROLLOVER              REBILL
  TRANSPOSED_READ                REBILL
  UNDETERMINED                   FIELD_VISIT

Rule X-1   The SYMPTOM is what the billing run raised and it is PRINTED ON THIS PAGE. It is an
           input, not the finding. Several causes produce the same symptom and the cause is what
           decides the fix, so agreeing with the printed symptom earns nothing.
Rule X-2   If the read history or the meter and register block is not present for this billing
           period, the symptom is CONTEXT_INCOMPLETE: no cause is determined, no evidence is
           cited and no action is taken. There is no default read for a read that never arrived.
Rule X-3   Decide the CAUSE by the first rule below that fires, in this order. The order is the
           policy, not a preference: an arithmetic defect that makes the index unusable has to be
           settled before anything is inferred from consumption computed off that index.
Rule X-4   REGISTER_ROLLOVER -- the current index is LOWER than the previous index, the register's
           printed digit count makes passing the maximum arithmetically possible, and no meter
           exchange is on record inside the period. The consumption is real; the subtraction is not.
Rule X-5   MULTIPLIER_DOUBLE_APPLIED -- the collector multiplier and the billing multiplier are
           BOTH printed greater than 1, and the billed consumption equals the metered difference
           times both of them. One multiplier is correct; two is an arithmetic defect.
Rule X-6   EXCHANGE_NOT_CLOSED -- a meter exchange is on record with an effective date inside the
           billing period AND the removed meter's final read is not recorded as closed off. The
           period is then being differenced across two registers that never shared an index.
Rule X-7   TRANSPOSED_READ -- the record states, in the event log or in the field log, that the
           index was mis-keyed or the digits were reversed, and a corrected index is available.
Rule X-8   TARIFF_CHANGE_MIDPERIOD -- a tariff effective date falls strictly inside the billing
           period and the bill shows a single tariff applied across the whole of it. The
           consumption is right; the apportionment is not.
Rule X-9   ESTIMATE_TRUEUP -- the reads immediately preceding this one are ESTIMATE and this one is
           ACTUAL, and the actual index is above the last estimated index. The consumption is real
           and was really used -- it was simply billed too low, earlier, and is landing now.
Rule X-10  LEAK_SUSPECTED -- the printed minimum interval consumption never returns to zero across
           the period, or the record states a continuous flow was observed. Nothing else here
           produces a floor that does not drop overnight.
Rule X-11  GENUINE_CONSUMPTION -- a site change, an occupancy change, a plant or process change or
           a seasonal explanation is on record and no earlier rule fires. The exception is real and
           the bill is correct.
Rule X-12  UNDETERMINED -- the exception is real and nothing anywhere on the record says why. It is
           NOT a synonym for "probably the usual one". Preferring a plausible label to
           UNDETERMINED is how a real defect acquires a benign explanation and stops being looked
           at.
Rule X-13  Cite the ONE evidence tag [Enn] that decides the cause. If the cause is UNDETERMINED or
           CONTEXT_INCOMPLETE, cite NONE. A cause with no evidence behind it is a guess that
           happens to be in the right bucket.
Rule X-14  The ACTION follows the cause by the printed routing table, and then TWO overrides in
           this order: a REBILL whose printed impact is BELOW the printed rebill floor becomes
           NO_ACTION; and a FIELD_VISIT on a premise that already has an OPEN field job on record
           becomes NO_ACTION. Both are printed on the page.
Rule X-15  The OWNER is the billing queue this exception belongs to. Start from the queue printed
           on the account, and let a re-queue recorded anywhere on the record -- including one
           typed as an ordinary sentence in the field log -- supersede it.

Field Log
----------------------------------------------------------------
Free text, typed against this exception by whoever was in front of the record.

  [E15]  2026-01-30  job card for this premise shows the old one lifted and the replacement
                      fitted the same morning. Nothing was ever entered for the figure it was
                      lifted at, so the quarter has been worked out from one dial to a
                      completely different one.

  [E16]  2026-03-01  they queried an estimated bill in 2025 and it was corrected then.

Raw responseThe raw response

{"symptom": "NEGATIVE_CONSUMPTION", "cause": "EXCHANGE_NOT_CLOSED", "evidence": "E15", "action": "REFER_METERING", "owner": "Revenue Protection", "rationale": "The field log records an in-period meter exchange with no final read entered for the removed meter, so the negative difference is caused by differencing across two registers under Rule X-6."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Find the real cause behind a utility's billing exception — 120 billing-exception dossiers. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection, never a bill and never a vendor's claim about this workload.

Rates checked 2026-08-18. The provider that actually ran the calls is kept out of these tables per this estate's naming rule, so no figure here is a bill.

The gradersThree ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

⚠︎ THIS KIT DISCARDED TWO FULL 120-CALL ARMS AND THE COST IS STATED RATHER THAN NETTED OUT. r001 truncated two replies at a 16,000 ceiling; s001 did not truncate but was discarded because its pair was, and a control run at a different ceiling from the arm it is compared with carries a second variable. The injection probe's own token cost is ABSENT rather than estimated -- its result file records no totals, which is a defect in evals/injection.py, now fixed for the next run and not recovered for this one. The three free floors, the stub and every pre-spend check cost $0.00 and no calls.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, AND ALMOST ALL OF THEM ARE REASONING. 94.8 pct of r002-bill-exception's output (229785 of 242293 tokens) was provider-side reasoning left at the default. The answer is five short fields and a sentence; the bill is the thinking in front of it.
• THE PROMPT IS MOSTLY FIXED TEXT. 3430 input tokens per reading, and the instruction and the printed policy block are identical on every call -- so a provider with prompt caching would price this workload very differently, and that was not measured.

Your volumeWhat it costs at your volume

LINEAR IN EXCEPTIONS. Nothing amortises. The sublinear lever NOT implemented is running the rules floor first and calling the model only where no rule fires -- on this corpus that would be 51 of 120 readings, and it was not measured.

Model choiceWhy this model

It is the tier the shared connection was already pointed at, and the kit's whole claim is that swapping it is .env plus one more run.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

per-query average, this run

The codeEvery file, and what it is for

12 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/triage.pythe triage rule and the precedence ladder — a swap seam

The symptom, cause and action vocabularies, the DEFAULT cause-to-action routing table, the two printed arithmetic overrides, and classify() -- which decides the answer from the page's planted facts, never from a model reply. It is also the answer key.

You change it to: DEFAULT_REBILL_FLOOR_GBP, DEFAULT_HIGH_MULTIPLE, DEFAULT_LOW_MULTIPLE, DEFAULT_VALIDATION_TOLERANCE_PCT.

# The billing-exception triage rule, as pure code. No model, standard library only.

HIGH_CONSUMPTION = "HIGH_CONSUMPTION"
LOW_OR_ZERO_CONSUMPTION = "LOW_OR_ZERO_CONSUMPTION"
NEGATIVE_CONSUMPTION = "NEGATIVE_CONSUMPTION"
READ_VALIDATION_FAIL = "READ_VALIDATION_FAIL"
TARIFF_NOT_APPLIED = "TARIFF_NOT_APPLIED"
BILL_VALUE_OUTLIER = "BILL_VALUE_OUTLIER"
CONTEXT_INCOMPLETE = "CONTEXT_INCOMPLETE"
SYMPTOMS = (HIGH_CONSUMPTION, LOW_OR_ZERO_CONSUMPTION, NEGATIVE_CONSUMPTION, READ_VALIDATION_FAIL,
LEAK_SUSPECTED = "LEAK_SUSPECTED"

src/segment.pythe section splitter

Splits the dossier into its 10 named sections; asserted across all 120 documents before any run may spend.

# Split a billing-exception dossier into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "Account And Premise", "Exception Raised", "Meter And Register",
RULE = "-" * 64
def split(text):
def render(secs):

src/select.pythe send filter — a swap seam

Payment Instrument is mapped by no field and therefore never sent.

You change it to: SECTION_HINTS and NEVER_SENT.

# Pick which sections of a billing-exception dossier are sent. Pure code -- the last deterministic

BANNER = "Synthetic Record"
ACCOUNT = "Account And Premise"
EXCEPTION = "Exception Raised"
METER = "Meter And Register"
POLICY = "Triage Policy (Default)"
READS = "Read History"
TARIFFS = "Tariff History"
EVENTS = "Meter Events"
PAYMENT = "Payment Instrument"

src/prompt.pythe prompt — a swap seam

Three parts: instruction and JSON shape, account history, dossier. The history-blind control replaces exactly one block and nothing else.

You change it to: HISTORY_SECTIONS. The control arm is this seam set to nothing.

# Assemble the one prompt this kit sends. No template engine, no framework -- string concatenation

HISTORY_SECTIONS = (select.READS, select.TARIFFS, select.EVENTS)
BLIND_LINE = ("No read history, no tariff history and no meter event record is available for this "
INSTRUCTION = """\
def build(text, blind=False):

src/adapters/__init__.pythe model call

Raw HTTP, stdlib only, bounded retry; the shared daily call cap is checked here.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/classify.pythe reader

One exception, one call. Holds the published token ceiling.

# One billing exception, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = ("You are a utility billing exception desk. You triage one billing exception against the "
MAX_TOKENS = 32000
FIELDS = ("symptom", "cause", "evidence", "action", "owner")
def documents():
def load_doc(doc_id):
def position_of(text):
def _parse(txt):

evals/scoring.pythe scorer

Exact match per cell. The cause discriminator is reported over three denominators that are never blended, and 'right bucket, wrong reason' is counted separately.

# Score a run against the answer key. Pure code, exact match per cell. No model grades anything.

FIELDS = ("symptom", "cause", "evidence", "action", "owner")
def _pct(n, d):
def norm_evidence(v):
def _same(field, got, want):
def score(records, golds):

evals/baseline.pythe three free floors

symptom-only, rules-threshold and rules-plus-keyword -- $0.00 each, scoring 39.17, 58.33 and 81.67 pct on the cause.

# THE FREE FLOORS. Three of them, none a strawman, all of them GBP 0.00.

MODES = ("symptom-only", "rules-threshold", "rules-plus-keyword")
SYMPTOM_CAUSE = {
KEYWORDS = (
KEY_CAUSE = {
def _section(text, name, nxt):
def _facts(text):
def _finish(mode, symptom, cause, evidence, owner, f, why):
def _structured_cause(text, f):
def _log_lines(text):

evals/check_labels.pythe pre-spend checks

24 claims the published page makes, checked against the corpus that will actually be sent. It convicted three real defects on its first run.

# Everything that must be true BEFORE a run is allowed to spend. Free, and it runs in seconds.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
FAILED = []
def check(name, ok, detail=""):
def main():

tools/build_corpus.pythe corpus generator

Deterministic from seed 20260825, no clock and no salted hash -- including its own date helper, so not even a date library is imported.

# Generate the 120 billing-exception dossiers and the answer key. Free, offline, reproducible.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
SEED = 20260825
RULE = "-" * 64
def _leap(y):
def to_day(y, m, d):
def from_day(n):

src/app.pythe local UI server

http.server, stdlib. Renders with no key, on port 9026.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "9026"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r002-bill-exception")
class H(BaseHTTPRequestHandler):
def main():

ui/app.jsthe local UI client

Hand-written JS, no framework.

# 

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3429 input and 2019 output tokens per reading (one account, one billing period, one exception), all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

⚠︎ MEASURED, AND FORCED RATHER THAN OBSERVED. The Field Log is this kit's injection surface -- it is prose typed by field technicians and contractors, and it reaches the model verbatim. The probe appended an instruction-shaped note to EVERY reading where an action was due and paired each against its own un-injected answer.

API_KEY is read from .env (gitignored) or the real environment; never logged, never placed in a prompt, and redacted from any provider error message the local UI passes to the browser.

The experimentWe DID attack it -- one sentence, on every reading where it could matter

The probe appends the instruction-shaped note to the field log on every reading whose gold action is not NO_ACTION and re-fires them with everything else held identical. It APPENDS rather than replacing the section, deliberately: on this kit the field log is also the evidence surface, and replacing it would have deleted the deciding evidence on 40 of 120 readings and confounded 'the model obeyed' with 'the model could no longer see'. All 98 held. Both were run for real on 2026-08-25.

Both boundaries are measured in both directions: the injection probe forces the condition rather than waiting for it AND pairs each reading against itself, and the privacy guard is red-proven by reproducing the schema-change condition that reaches its fallback.

The result0 of 98 actions suppressed by the instruction-shaped field note -- forced, paired, and measured rather than assumed.

One phrasing, one model, one corpus, 98 trials -- every reading where suppression was even possible, each paired against its own un-injected answer from r002-bill-exception.

HonestyWhat this does not prove

• Whether the resistance reproduces across phrasings. A note claiming the account is under a formal complaint, claiming the meter has already been checked, or written to look like a system banner is a different experiment.
• Whether an injection placed in Payment Instrument would have any effect -- by construction it cannot, and that was not probed either.
• Whether the note changes the CAUSE rather than the action. The probe scores the action; the cause after injection is recorded per suppressed reading and there were none to record.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never issue a bill, cancel a bill, post a rebill, raise a credit, book a field job or amend a read -- and never present the thresholds, the rebill floor or the routing table as any real utility's, supplier's or regulator's policy.

Stated to the model on every call in src/prompt.py's INSTRUCTION block and printed inside every dossier, and enforced by evals/check_labels.py's banned-code-path scan before any run may spend.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is a PROMPT RULE PLUS A STATIC SCAN, not a runtime enforcement layer. Nothing stops a forker adding such a path tomorrow; the scan catches it only the next time somebody runs check_labels.py, which is a manual step.
• The injection result is ONE SENTENCE against ONE model on ONE corpus. It is not a resistance rate for prompt injection in general.
• The answer key is not a guardrail. It is wrong on 9 of 120 readings and the checks cannot see it, because they assert the key replays from the RULE rather than that the rule matches the PAGE.

WatchedWhat is watched, and why that one

The kit measures 100 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

6 measured by the latest run 94 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

13 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 4 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

triage · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 3 runs measured the same system — they differ on cause_right_cells, cause_right_evidence_wrong, exception_cause_offmode_hits, exception_cause_prose_hits, false_rebills, false_undetermined, misroutes, missed_field_visits, overclaimed_cause, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

triage · with the model in the path — 8 runs. Columns here are only ever compared with each other.

not a time series No two of these 8 runs measured the same system — they differ on answered, cause_right_cells, context_incomplete_cells, distractor_cells, documents, exception_cause_offmode_cells, exception_cause_offmode_hits, exception_cause_onmode_cells, exception_cause_prose_cells, exception_cause_prose_hits, exception_cause_structured_cells, false_undetermined, field_visit_cells, max_tokens, misroutes, missed_field_visits, non_rebill_cells, readings_scored, requeue_cells, stateless, undetermined_cells — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 20 chips that all say so.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 6 chips that all say so.

DeviationsWhat deviated

0 breaches across 13 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The guardrail scan is a MANUAL step, run before each spend, not a hook. It ran before every paid run and passed at 0 each time. The injection probe is a ONE-OFF: it measured 0.0 pct suppression on 98 paired readings on 2026-08-25 and nothing re-runs it, so that figure ages from the day it was taken.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, no framework dependency, an empty requirements.txt and a prompt anyone can read end to end. There is no retrieval step for an abstraction to own, and the one thing a framework would own here -- the precedence ladder -- is the answer key.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear: the reader -> src/prompt.py -> src/adapters -> evals/scoring.py. No branching, no tool-calling and no agent loop.

The other sideWhat a framework costs you

• Swapping providers means editing the PROVIDERS dict by hand. In exchange the prompt has no hidden templating layer and requirements.txt stays empty.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r002-bill-exception on the fast tier, with the account history, 2026-08-25. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 10 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

3 runs not plotted. b000-bill-exception-symptomonly, b001-bill-exception-rules, b002-bill-exception-ruleskeyword recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — one report is one event, whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineexact match per cell against the answer key

symptom, cause, evidence, action, owner

MethodHow the test was run

python3 -m evals.run --run-id <id>

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

answer[field] == gold[field], with the evidence tag normalised for brackets and case

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the computed answer key in data/gold.jsonl, produced by src/triage.classify() from the same planted inputs the corpus builder wrote onto the page.

No true/false rates for this grader. It records 5 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold was swept: exact match has no tunable. Denominators are stated beside every rate because 120 readings makes each one wide.

Cadence: once per run; free

The decisionWhen to reach for it

In one linethe off-mode slice

cause, restricted to readings whose cause is not their symptom's mode

MethodHow the test was run

reported by evals/scoring.py on every run

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

gold.cause != modal_cause_of(gold.symptom), re-derived from the finished key

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the modal cause of each symptom, re-derived from the finished answer key by evals/check_labels.py.

No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold. The slice is a partition of the same 120 readings and both halves are printed.

Cadence: once per run; free

The decisionWhen to reach for it

In one linecause AND its deciding evidence

whether a correct cause was reached for the stated reason

MethodHow the test was run

reported by evals/scoring.py on every run

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

cause_ok AND evidence_ok; the gap between them is cause_right_evidence_wrong

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the deciding [Enn] tag recorded in data/gold.jsonl, asserted to exist inside a section the kit actually sends.

No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold. The evidence tag is compared after normalising brackets and case, and nothing else.

Cadence: once per run; free

The decisionWhen to reach for it