The business caseThe problem this solves

'Which accounts are unbilled beyond cycle' is a SELECT and nobody needs a language model for it. The question a billing operation actually has to answer is the next one: how close is each of them to the point where the charge cannot be recovered at all, and how much of this month's unbilled balance is already past it. That is not the elapsed time since the last bill -- on all 150 statements here the last bill is an estimate issued weeks ago that moves nothing. It is the time since the oldest UNBILLED ACTUAL meter read, minus every stretch in which the delay was the customer's own, on an account still inside the protection, judged against the remediation lead time for its blocker. Three of those four terms can sit in a cycle whose statement is not on your screen. Somebody working an unbilled-account queue by hand: opening each account, finding the oldest read that was actually billed rather than the last bill issued, going back through earlier cycles' exception entries to work out how much of the delay was the customer's own, and only then deciding whether this is an account to chase this week or one the charge is already lost on -- and totalling the lost ones for the month-end pack.

The inputThe actual cycle statements

The corpus is 150 cycle statements, 0.53 MB (json 1 · jsonl 1 · txt 150). AN UNBILLED-ACCOUNT WATCHLIST IS A LIST OF HOUSEHOLDS. Not contracts -- homes: who lives there, where the meter is, how much energy they have used that nobody has billed them for, and an agent's note about why. There is no public corpus of that and there never will be one, for the same reason there is no public corpus of anyone's bank statements; a supplier's own unbilled book is among the most sensitive operational files it holds, because it joins consumption to identity to money owed. Generating it also bought the one thing a captured corpus cannot give: the answer key is src/backbill.replay's output over the planted patterns, so an interval rule this fiddly cannot carry its author's misreading into the score. ⚠︎ AND IT SHIPS BLOCKED PENDING ANCHOR. BACKBILL_WINDOW_DAYS = 365, all six remediation lead times, the segment threshold and the dispute rule are INVENTED for this kit. They reproduce no statute, no regulator's guidance, no licence condition, no industry code and no supplier's billing policy, and name none. The catalogue row this kit was built from records the real window value as an OPEN QUESTION shared with a sibling row -- 'one clock, not two independently answered ones' -- and nobody here has confirmed it. The rule text is reproduced in full on all 150 pages precisely so it can be read, disbelieved and replaced.

Synthetic Record
----------------------------------------------------------------
This file is synthetic. It was generated for an open evaluation kit and describes no real
customer, household, address, meter, supplier or charge. Account ACC-0001, cycle C1.

Account
----------------------------------------------------------------
Account       : ACC-0001
Segment       : domestic (inside the back-billing protection)
Product       : Prepayment converted to credit, electricity
Meter type    : smart, non-communicating
Supply since  : 2020-05-14
Billing cycle : monthly, cycle end on day 20

Back-Billing Rule
----------------------------------------------------------------
Back-billing window : 365 days
Remediation lead time published for each blocker:
  EXCEPTION_HOLD   14 days
  STUCK_ORDER       7 days
  METER_PROBLEM    45 days
  NO_ACCESS        60 days
  READ_DISPUTE     30 days
  CAUSE_UNKNOWN    30 days

  1. A charge for energy consumed more than 365 days before the date a valid bill is issued cannot
     be recovered from the customer and must be written off.
  2. The 365 days run from the date of the OLDEST UNBILLED ACTUAL METER READ. They do not run from
     the last bill issued, and they do not run from the date the billing exception was raised.
  3. The window is SUSPENDED only for a period the Exception Log records as a customer-attributable
     suspension -- access to the meter refused or not arranged, or a reading the customer undertook
     to provide and did not. A suspension runs from the day it is OPENED up to but not including the
     day it is CLOSED, and days inside it do not count towards the window. A standing block whose
     recorded reason sounds customer-attributable does NOT suspend the window unless a suspension
     was opened for it.

The outcomeWhat a good result looks like

Per cycle: the proximity band after the suspension rule has been applied, whether the back-billing clock is RUNNING, SUSPENDED or NOT_PROTECTED, how many days now count towards the window, and which blocker is standing -- plus a four-field carried accrual the next cycle is judged against, written by code from the parsed dates and never from the model's reply.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Sorting an unbilled queue where nothing was ever suspended and every account's history is on its own page — the free one-page floor, at $0.00
  The floor gets the band right on 86.67 pct of all cycles and the blocker code on 98.0 pct -- BETTER than the model's 91.33. On the clean-drift, suspension-inside-one-cycle and not-protected-by-segment patterns it is exactly right, because the elapsed calendar IS the risk clock there.
• An account whose suspension began in a cycle you are not holding -- a CLOSED entry with no OPENED, or an empty exception log — the fast tier WITH the carried accrual
  100.0 pct of the bands and 92.0 pct of the day counts on the 25 memory-dependent cells, against the floor's 20.0 pct and 0 pct. This is the half the accrual plainly buys, and it is the difference between AT_RISK at 360 days and BREACHED at 396 on the same statement.
• Producing the month-end unbilled-exposure figure for finance — the fast tier WITH the carried accrual, with the per-account list published beside the total
  The total is exact ($29,024.35 of $29,024.35, 0.0 pct error) where the floor is 49.39 pct OVER and the stateless arm 65.98 pct UNDER. ⚠︎ But publish the list too: this grader is where errors cancel, and a total can be right while rows are wrong.
• Deciding to write a balance off — a person, reading the band, the day count and the carried accrual the kit printed
  The highest-consequence output here is BREACHED, and both no-memory arms get the total wrong in opposite directions -- the floor by 49.39 pct over, the stateless arm by 65.98 pct under. Nothing in this kit writes anything off and nothing should be added that does.

And where nothing here is good enough:

• Deciding which exception code an account's recorded reason belongs to — neither, as built -- free code beats the model and both are worse than a curated list
  The floor scores 98.0 pct and the model 91.33 pct (96.0 pct under the most generous reading of its own answers). The model invented ten CAUSE_UNKNOWNs on reasons that map cleanly, while getting 18 of 18 genuinely unrecorded ones right. And the catalogue row already records the real taxonomy as unconfirmed, so this field is measuring agreement with an invented list.
• An account with years of history rather than three cycles — nothing here yet -- measure it first
  The accrual is four scalar fields, so the COST does not grow. The accuracy is unmeasured: every rule in this corpus resolves inside three cycles, and the one error shape this run has (the half-open boundary, one day out) is exactly the shape a longer history gives more chances to accumulate -- on a RUNNING TOTAL, one day out per cycle is forty days out after forty cycles.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-23. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-23 — r001-unbilled-watch. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with NO key configured reproduces the corpus byte-identically (python3 tools/build_corpus.py), passes every pre-flight assertion (python3 -m evals.check_labels), scores the free one-page floor (python3 -m evals.run --run-id b000-unbilled-watch-onepage --baseline), re-scores the recorded run under the other reading of one blocker code (python3 -m evals.ambiguity) and serves the whole UI (python3 -m src.app). Five commands, plain python3, no install step -- requirements.txt names nothing because the kit imports nothing outside the standard library. What it cannot do without a key is re-run the scored eval or its stateless control.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per cycle: one HTTP request carrying the assembled prompt, and the reply parsed to a band, a clock status, a day count and a blocker code. Splitting the statement into sections, dropping the one section no field asks for, parsing the dates and advancing the accrual all happen outside this clock and cost no network at all. THE TAIL IS THE STORY: p95 (55.9s) is 4.5 times p50 (12.3s). Output length here is driven by how hard the INTERVAL ARITHMETIC is, not by how long the page is -- every statement is within 6 pct of every other in size and replies ran from 282 to 12664 tokens, a 44.9x spread.

Current processWhat it replaces

Somebody working an unbilled-account queue by hand: opening each account, finding the oldest read that was actually billed rather than the last bill issued, going back through earlier cycles' exception entries to work out how much of the delay was the customer's own, and only then deciding whether this is an account to chase this week or one the charge is already lost on -- and totalling the lost ones for the month-end pack.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptArchitectureArchitectureApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• 3 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one accounts (3 cycles each) is

No split, and no chunking. The unit is an ACCOUNT -- a run of three consecutive monthly cycle statements processed strictly in order, because cycle 3's prompt contains an accrual produced by cycle 2. Each statement goes whole into one call, minus the one section no field asks for.

SetupWhat the setup figure measured

There is no index and no retrieval step. tools/build_corpus.py writes 150 documents and the answer key from a fixed seed with no clock read and no model called; nothing is embedded, ranked or cached.

LicenceLicence

MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved; every account, tariff, meter, reading, exception entry, household name and address is invented here. Verified against the repository's own LICENSE file on 2026-08-23.

Bring your ownBring your own cycle statements

Replace src/backbill.py FIRST, and not as an option. It is your regulator's window and your operation's lead times, the shipped ones are invented, and data/gold.jsonl is literally its output -- change the rules and re-run tools/build_corpus.py and the answer key follows. Then point the corpus generator at your own accounts, or delete it and drop real cycle statements into data/corpus/ named <ACCOUNT>-C<n>.txt with the same seven section headings. src/segment.py, src/select.py, src/prompt.py and src/state.py read sections by name and do not care where the statements came from.

⚠︎ And what stops being true when you do: Every figure here is a property of a three-cycle corpus of 50 invented households, measured once. Three of the denominators are small: 11 AT_RISK cells, 25 memory-dependent cells and ONE false alarm -- one row is not a rate. And the headline hides its own shape: 99.33 pct band accuracy is 149 correct cells and ONE mis-mapped blocker, so a book with a different mix of recorded exception wording moves that figure without the model having changed at all. ⚠︎ The thing most likely not to survive contact with a real book is the blocker taxonomy, which is the field this run is WORST on and the one the catalogue row already records as unconfirmed.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 6,182 characters assembled, 2,491 are instructions — 40% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Replayed from the kit's own src/prompt.py::build on ACC-0001-C1 with the exact carried accrual r001-unbilled-watch recorded for that call ({"days_at_risk": 0, "prev_band": null, "protected": true, "suspension_open": false}) -- the identical code path the run used, not a paraphrase. THE SPLIT IS IN CHARACTERS AND THE PAGE SAYS SO: per-part TOKEN counts were not measured, because measuring them means sending nested prefixes of the prompt to the provider and this kit spent no calls on it. The billed totals (222,929 in / 306,959 out over 150 calls) are measured; nothing here apportions them across the parts. The stateless control's build of the same statement differs only inside the carried-state block, and evals/check_labels.py asserts every other byte is identical -- that is the entire experiment.

Full promptThe prompt in full

You apply a written back-billing rule to one unbilled energy account for one cycle. You answer with one JSON object and no other text.

You are reviewing one energy account that is unbilled beyond its billing cycle, for one cycle, and
deciding how close it is to the back-billing limit written in the statement below.

The back-billing rule, its window length and the remediation lead time for every blocker are
reproduced in the statement. Apply them exactly as written. The rule depends on how many days of
this account's unbilled period have ALREADY been counted towards the window in EARLIER cycles, which
you cannot see; what is known is stated under "Carried state" and is the only history available to
you. Do not assume anything about earlier cycles beyond it.

How to count:
 - Days counted in this cycle are stated in the Billing Position.
 - Subtract from them every day inside a customer-attributable suspension the Exception Log records.
   A suspension runs from the day it is OPENED up to but not including the day it is CLOSED. A
   standing block whose reason sounds customer-attributable does NOT suspend the window unless a
   suspension was opened for it. The Exception Log shows only the entries raised in the period this
   statement covers; a suspension the carried state says was already open has been running since
   before this cycle and stops on the day this statement records it closing.
 - Add what is left to the days the carried state says were already counted.
 - An account outside the protection has no window at all: answer band NO_LIMIT, clock_status
   NOT_PROTECTED, and days_at_risk 0.

Answer with a single JSON object and nothing else:

{"band": "WITHIN_LIMIT|APPROACHING|AT_RISK|BREACHED|NO_LIMIT",
 "clock_status": "RUNNING|SUSPENDED|NOT_PROTECTED",
 "days_at_risk": <whole number of days now counted towards the window>,
 "blocker_cause": "EXCEPTION_HOLD|STUCK_ORDER|METER_PROBLEM|NO_ACCESS|READ_DISPUTE|CAUSE_UNKNOWN",
 "rationale": "one sentence, naming the rule you applied and the days you counted"}

"band" is the band AFTER the suspension rule has been applied, which is not the band the elapsed
time since the last actual read alone would imply. "clock_status" is SUSPENDED only if a
customer-attributable suspension is still open at this cycle end. "blocker_cause" is the standing
billing block mapped to one of the six codes above; use CAUSE_UNKNOWN when the recorded reason maps
to none of the other five rather than choosing the nearest.

Carried state
----------------------------------------------------------------
No earlier cycle has been recorded for this account. This is its first appearance on the watchlist, so the days already counted towards the window are only those shown on this statement.

Cycle statement
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
This file is synthetic. It was generated for an open evaluation kit and describes no real
customer, household, address, meter, supplier or charge. Account ACC-0001, cycle C1.

Account
----------------------------------------------------------------
Account       : ACC-0001
Segment       : domestic (inside the back-billing protection)
Product       : Prepayment converted to credit, electricity
Meter type    : smart, non-communicating
Supply since  : 2020-05-14
Billing cycle : monthly, cycle end on day 20

Back-Billing Rule
----------------------------------------------------------------
Back-billing window : 365 days
Remediation lead time published for each blocker:
  EXCEPTION_HOLD   14 days
  STUCK_ORDER       7 days
  METER_PROBLEM    45 days
  NO_ACCESS        60 days
  READ_DISPUTE     30 days
  CAUSE_UNKNOWN    30 days

  1. A charge for energy consumed more than 365 days before the date a valid bill is issued cannot
     be recovered from the customer and must be written off.
  2. The 365 days run from the date of the OLDEST UNBILLED ACTUAL METER READ. They do not run from
     the last bill issued, and they do not run from the date the billing exception was raised.
  3. The window is SUSPENDED only for a period the Exception Log records as a customer-attributable
     suspension -- access to the meter refused or not arranged, or a reading the customer undertook
     to provide and did not. A suspension runs from the day it is OPENED up to but not including the
     day it is CLOSED, and days inside it do not count towards the window. A standing block whose
     recorded reason sounds customer-attributable does NOT suspend the window unless a suspension
     was opened for it.
  4. The protection does not apply to an account outside the protected segment, nor to an account
     where a valid bill was issued and the customer has disputed it. Those accounts have NO LIMIT
     and are not on this watchlist.
  5. An account is AT RISK when the days left before the window closes are fewer than the published
     remediation lead time for its blocker; APPROACHING when they are fewer than twice that lead
     time; BREACHED when the window has already closed.

Billing Position
----------------------------------------------------------------
Cycle                         : C1
Cycle end                     : 2026-06-20
Previous cycle end            : none - this is the first cycle this account appears on the watchlist
Days counted in this cycle    : 153
Oldest unbilled actual read   : 2026-01-18
Last bill issued              : 2026-04-27 (estimated - does not close the unbilled period)
Unbilled consumption estimate : 1,637 kWh
Unbilled value estimate       : 579.20 USD

Exception Log
----------------------------------------------------------------
Standing billing block : BLK-71935, raised 2026-02-15, still open
  reason as recorded   : "meter room locked and no arrangement has been made to open it"
Entries raised in the period this statement covers. A suspension is a separate, dated episode and
does not change the standing block reason above. Entries from earlier statements are not reproduced
here.
  (no exception entries were raised in the period this statement covers)

Operational Notes
----------------------------------------------------------------
This account was picked up in the unbilled sweep for this cycle. Nothing has been billed and
nothing has been written off.

Raw responseThe raw response

{"band":"WITHIN_LIMIT","clock_status":"RUNNING","days_at_risk":153,"blocker_cause":"NO_ACCESS","rationale":"Applying the 365-day back-billing window with no customer-attributable suspension recorded, the 153 days counted this cycle leave 212 days remaining, which is within the 60-day NO_ACCESS remediation lead time."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch unbilled power accounts before the charge expires — 150 cycle statements. One model answered, and every answer was then graded Five different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

There is no LLM-as-judge in this kit and that is not a shortcut. Three of the four answered fields are closed sets -- five bands, three clock states, six blocker codes -- and the fourth is a whole number of days. A judge is for answers whose correctness is a matter of reading; these are matters of equality and arithmetic, and asking a model to grade 'AT_RISK == AT_RISK' would add cost, variance and a second thing to be wrong. evals/scoring.py compares each cell exactly against a gold set that src/backbill.replay computed, and scores the stateless control and the free floor through the same function.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran all 309 calls is kept out of these tables per this estate's naming rule, so nothing here is what was paid. The real spend for this kit's whole build is recorded in the shared call ledger, not on this page.

Grading unitWhat the grading figure prices

Grading calls nothing. A measured $0.00, not an unpriced one -- and evals/ambiguity.py re-scores the whole run under the other reading of one blocker code for $0.00 as well, because it reuses the recorded replies.

The gradersFive ways to grade

READ THIS ROW BEFORE THE STATELESS CONTROL, BECAUSE IT IS THE STRONGER NO-MEMORY ARM AND ON TWO FIELDS IT BEATS THE MODEL. It is not a straw man: it already avoids the mistake rule 2 exists to warn about (it anchors on the oldest unbilled ACTUAL read, not the last bill), it subtracts every suspended day the page shows, and its keyword table tests 'reason not recorded' BEFORE the topical words, which is the most generous reading of a keyword matcher available. It gets the blocker code right on 98.0 pct of cells against the model's 91.33, and it raises no missed watchlist cycles at all. What it cannot do is the three shapes that need memory: a CLOSED entry with no OPENED, an empty exception log hiding a still-open suspension, and a dispute raised in an earlier cycle. ⚑ AND ITS EXPOSURE ERROR RUNS THE OPPOSITE WAY TO THE STATELESS MODEL'S. The floor over-states the balance beyond the window by 49.39 pct ($43,358.99 against a true $29,024.35); the stateless model UNDER-states it by 65.98 pct ($9,873.05). Two arms with the same information, erring in opposite directions -- which means 'no memory' is not a bias anybody could correct for with a constant.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same cycle, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

309 live calls are on the shared ledger for this kit and every one of them is behind a figure on this page: 9 calibration (c000, at a 32,000-token ceiling on the three hardest accounts), 150 scored, 150 the stateless control. NOTHING WAS DISCARDED and no run was re-fired -- the corpus defects this kit found (see Data.breaks_on) were both caught by reading generated documents BEFORE any call was made, which is the whole argument for reading your own corpus first. Everything else -- the free one-page floor, the wiring stub, the scorer, the pre-flight and the ambiguity re-score -- is pure code and costs $0.00. Figures are projected onto the Google Gemini 3 Flash card, not the real spend.

Cost driversWhat actually moves the bill

• PROVIDER-SIDE REASONING, 95.3 pct of output tokens (292,625 of 306,959) and therefore most of the bill. It was left at the provider's default and this kit has not measured what turning it off does to the answers.
• HOW HARD THE INTERVAL ARITHMETIC IS, NOT HOW LONG THE PAGE IS. Every statement is between 3,641 and 3,869 bytes -- within 6 pct of each other -- and output tokens ran from 282 to 12,664, a 44.9x spread on a corpus of one shape. The long replies are the boundary-suspension cycles.
• WHETHER THE PROMPT CARRIES THE ACCRUAL -- AND IT IS A SAVING, NOT A COST. One short sentence adds a handful of input tokens a cycle and CUTS the output by 2.32x, because a model with the accrual has an addition to do and a model without one has a puzzle it cannot solve.
• THE FIXED HEAD OF THE PROMPT. The system message and the instruction are 2,491 of 6,356 characters -- 39 pct, byte-identical on all 150 calls. Add the back-billing rule block, which varies in nothing but nothing, and the account's own numbers are the smaller half of what is sent.

Your volumeWhat it costs at your volume

Linear in CYCLES, and flat in history length, which is the unusual half. Each cycle is one call whose input is one statement plus four carried fields, so an account on its fortieth cycle costs exactly what one on its third costs. 150 cycles project to $1.0323 on the shared rate card, so ten times the set is about $10.32. ⚠︎ AND A REAL UNBILLED BOOK IS THE PLACE THIS ARITHMETIC STOPS BEING COMFORTABLE: at $0.006882 a cycle, a hundred thousand unbilled accounts checked monthly is about $688 a month, every month, and the free one-page floor already gets 86.67 pct of the bands right for nothing. The honest reading is that this belongs on the subset the floor cannot answer, not on the whole book. What does NOT scale is wall clock: cycles within one account are strictly serial.

Model choiceWhy this model

It is the tier the shared key is pointed at, and it is the cheap one -- the right place to start a question whose honest answer might be 'free code already does most of this', which on the blocker-code field it does, better than the model.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every stateful number on these pages: 150 cycles, one completion call each, one tier. The 150-call stateless control and the 9 calibration calls are accounted for separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

14 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pythe corpus generator — a swap seam

Generates 50 accounts x 3 consecutive monthly cycles = 150 statements from a fixed seed (SEED = 20260823), across two fuels and both segments. It plants eight patterns, of which 20 accounts carry a suspension and 9 of those span a cycle boundary -- the trap. Consumption, unit rates and standing charges are drawn PER FUEL; see Data.breaks_on for the defect that rule was written from. The gold labels are src/backbill.replay's output, never typed.

You change it to: Point it at your own accounts, tariffs and exception vocabulary, or delete it and drop real cycle statements into data/corpus/ named <ACCOUNT>-C<n>.txt with the same seven section headings. Everything downstream reads statements by account and cycle and does not care where they came from.

# Generate the synthetic unbilled-account corpus and its computed answer key.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
SEED = 20260823
RULE = "-" * 64
CYCLE_MONTHS = [(2026, 6), (2026, 7), (2026, 8)]
PATTERNS = [("CD", 8), ("SE", 9), ("SO", 6), ("SL", 5),
PRODUCTS = [

src/backbill.pythe back-billing arithmetic — a swap seam

The window rule as pure code: days accrued, minus dated customer-attributable suspensions, against the remediation lead time for the standing blocker. No model, no judgement. ⚠︎ The window length and every lead time in it are INVENTED and the kit ships that as a stated blocker -- see Data.why_this_corpus.

You change it to: BACKBILL_WINDOW_DAYS, LEAD_DAYS and step() are your regulator's window and your operation's lead times. Change them there and re-run tools/build_corpus.py, which recomputes the whole answer key from the same function -- the gold cannot drift from the rules because it is their output. ⚠︎ THIS IS THE FIRST FILE TO REPLACE, not an optional one: the shipped window is invented.

# The back-billing window, and the arithmetic it implies. Pure code, no model.

BACKBILL_WINDOW_DAYS = 365
WITHIN_LIMIT = "WITHIN_LIMIT"
APPROACHING = "APPROACHING"
AT_RISK = "AT_RISK"
BREACHED = "BREACHED"
NO_LIMIT = "NO_LIMIT"
BANDS = (WITHIN_LIMIT, APPROACHING, AT_RISK, BREACHED, NO_LIMIT)
RUNNING = "RUNNING"
SUSPENDED = "SUSPENDED"

src/state.pythe carried accrual — a swap seam

SEAM 2 -- the thing that makes this a monitor. Four fields per account (days already counted, whether a suspension is open, the band last reported, whether it is still protected), written from the arithmetic and never from the model's reply, and rendered into one English sentence for the prompt. It costs the same on cycle 40 as on cycle 3.

You change it to: load()/save() are a JSON file today. Point them at a table, a key-value store or a billing data mart and nothing else in the kit changes -- for_account() and advance() are the whole interface, and describe() is the only thing the prompt sees.

# The carried state -- the thing that makes this a monitor and not another one-shot kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEFAULT_PATH = os.path.join(HERE, "data", "state.json")
def load(path=DEFAULT_PATH):
def save(store, path=DEFAULT_PATH):
def for_account(store, account_id):
def advance(store, account_id, elapsed, suspended, susp_open_end, cause, protected):
def describe(state):

src/segment.pythe section splitter

Splits a cycle statement into its seven named sections. Pure code; evals/check_labels.py asserts all seven are present in all 150 documents before a run may spend.

# Split a cycle statement into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "Account", "Back-Billing Rule", "Billing Position",
RULE = "-" * 64
def split(text):
def render(secs):

src/select.pythe section selector — a swap seam

Decides which sections reach the model. Customer Contact -- the household's name, supply address, phone and email -- is mapped by no field and is subtracted unconditionally by _fallback(), so the one section whose subject is a PERSON never leaves the machine even when a hint matches nothing.

You change it to: SECTION_HINTS maps a field to the sections it needs; NEVER_SENT names what is withheld whatever happens. ⚠︎ On a domestic book this is the load-bearing seam: check NEVER_SENT covers every section your statements carry that names a person, before anything spends.

# Pick which sections of a cycle statement are sent. Pure code -- the last deterministic step

BANNER = "Synthetic Record"
ACCOUNT = "Account"
RULEBLOCK = "Back-Billing Rule"
POSITION = "Billing Position"
EXCEPTIONS = "Exception Log"
CONTACT = "Customer Contact"
NOTES = "Operational Notes"
NEVER_SENT = (CONTACT,)
SECTION_HINTS = {

src/prompt.pythe prompt

Three parts: the fixed instruction, the carried-accrual sentence, and the selected sections in document order. The stateless build replaces the accrual sentence with a line saying no history is available -- byte-identical everywhere else, which is what makes the control a control.

# Assemble the one prompt this kit sends. No template engine, no framework -- string concatenation

INSTRUCTION = """\
def build(text, carried, stateless=False):

src/adapters/__init__.pythe model adapter — a swap seam

SEAM 1 -- the model. Raw HTTP for every provider, standard library only. Returns input/output tokens, reasoning tokens and finish_reason beside the text, and separates a transport failure from an HTTP status so a dropped connection is retried rather than recorded as a lost document.

You change it to: .env plus the same run again. Any OpenAI-compatible provider or Anthropic.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/monitor.pythe watch

One cycle, one call. Parses the cycle dates and the unbilled value off the page with a regex (the model is never asked to read a date), assembles the prompt, calls the model, parses the reply. Holds MAX_TOKENS = 24000, the published ceiling.

# One cycle, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = ("You apply a written back-billing rule to one unbilled energy account for one cycle. "
MAX_TOKENS = 24000
FIELDS = ("band", "clock_status", "days_at_risk", "blocker_cause")
def documents():
def accounts():
def load_doc(doc_id):
def position_of(text):

src/app.pythe local UI

One account, one cycle, its carried state and its proximity, on 127.0.0.1:9004. Renders with no key. It shows the carried sentence verbatim and the free one-page floor's answer beside the model's, so a reader can see on which rows the model bought nothing.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
PORT = int(os.environ.get("PORT", "9004"))
def _gold():
GOLD_ROWS = _gold()
def carried_for(doc_id):
class H(BaseHTTPRequestHandler):
def main():

evals/baseline.pythe free one-page floor

Everything a reader of ONE statement can compute: the oldest unbilled actual read, the suspended days this page shows, an ordered keyword map of the standing reason, and the segment line. 0 calls, $0.00, scored through the identical scorer.

# The free, no-model floor: the whole answer, computed from ONE statement, with no memory.

KEYWORDS = [
def _date(s):
def classify(reason):
def read_page(text):
def review(text):

evals/scoring.pythe scorer

Exact match per cell against the computed gold, split five ways an average would hide: the four fields, the cycles on the watchlist, the quiet cycles, the memory-dependent subset, and the portfolio exposure in dollars. No judge model.

# Score a run against the computed gold. Pure code, no model, no judge.

FIELDS = ("band", "clock_status", "days_at_risk", "blocker_cause")
QUIET_BANDS = ("WITHIN_LIMIT", "NO_LIMIT")
def _pct(n, d):
def _norm(v):
def _as_int(v):
def score(records, golds):
def compare(stateful, stateless):

evals/check_labels.pythe pre-flight

Everything that must be true before a run may spend: 150 documents, 50 gap-free accounts, every document declaring itself synthetic on line 1, gold reproducible from backbill.replay, no statement naming another cycle, the two prompt builds differing in exactly one block, the privacy guard red-proven in both directions, per-fuel consumption inside a household's range, and no code path anywhere that releases a bill or writes a charge off.

# Everything that must be true before a run may spend. Free, no key, no model.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
FAILS = []
def check(label, ok, detail=""):
def _date(s):
def main():

evals/ambiguity.pythe ambiguity re-score

Re-scores the recorded run against the OTHER defensible reading of one blocker code. Free, no new calls. It exists because ten of the thirteen blocker misses are one recorded sentence whose code name does not describe it, and the honest response is to publish both numbers rather than reword the taxonomy after seeing the answers.

# Re-score the recorded run under the OTHER defensible reading of one blocker code. Free, no calls.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
DISPUTED_REASON = "the customer undertook to send a reading and has not done so"
ALSO_ACCEPTED = B.CAUSE_UNKNOWN
def reason_of(doc_id):
def main(run_id="r001-unbilled-watch"):

evals/run.pythe run harness

50 account chains, three strictly-ordered cycles each, 12 concurrent workers. The accrual advances even for a cycle whose CALL failed, so one transport error cannot turn into three scored failures.

# Run the watch over the 50 accounts and write one result file. THIS SPENDS MONEY.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
def load_gold():
def stub_complete(cfg, system, user, max_tokens=1024):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 1486 input and 2046 output tokens per cycle, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

No untrusted external input surface in the shipped version. Every field that reaches the prompt comes from tools/build_corpus.py's fixed seed, including the Operational Notes, which are one of three fixed sentences. src/select.py names the notes as the field an outside party WOULD influence in a real deployment -- prose a billing agent types into an exception queue -- and sends them deliberately, so the surface is visible rather than hidden. On this corpus nobody outside the repository wrote a byte of it.

The provider key lives only in .env (gitignored) or the shared repo-root .env, read by src/config.py and never logged. config.sources() reports WHICH file a value came from and never the value. config.save() opens the file 0600 before the write rather than chmod-ing it after, so the key never exists at the default umask even for an instant. src/app.py redacts the key and the base URL out of any provider error before it reaches the browser.

The experimentWe did not attack it -- and the boundary that matters most on this vertical was proven by breaking it on purpose, twice

An indirect prompt injection needs a field an outside party controls that reaches the prompt, and this corpus has none. So the five rows above are boundaries, not payloads. One of them was red-proven: the privacy guard was replaced with the naive fallback every sibling kit once shipped and all 150 statements leaked the household's name, supply address, phone number and email, then it was restored and all 150 held. ⚠︎ It took two attempts to do that honestly. The first red-proof swapped the guard and measured ZERO leaks -- which looked like the guard working and was the test never firing, because the fallback is unreachable on today's corpus. A guard that has never been MADE to fail is a guard nobody has tested, and a red-proof that cannot fail is worse, because it reports a green. Confirmed by assertion and by reading the recorded run, not by an attack trial, on 2026-08-23 -- no attack run was fired; see redteam.why.

Each boundary above was checked by running an assertion or by reading the recorded run, not by an attack trial -- there is no untrusted field on this corpus to construct a payload against. One of the five is red-proven, meaning the guard was removed and the failure observed rather than merely asserted while passing, and its first red-proof was itself wrong and is recorded as such.

The result0 attack trials, five boundaries checked -- and the privacy boundary red-proven by removing the guard and watching all 150 statements leak a household's name and supply address.

The Operational Notes ARE the field an outside party would influence in a real deployment, and this kit sends them rather than hiding them -- but on this corpus they are one of three sentences chosen by a seeded generator, so there is nothing adversarial in them to catch. A version pointed at real agent prose reopens the question and should be attacked before it ships. What IS measured is the privacy boundary, in both directions, before every run.

HonestyWhat this does not prove

• Whether a real deployment's Operational Notes -- prose a billing agent writes freely -- would carry an instruction the model follows. Not applicable to the shipped corpus, and the first thing to attack if this is pointed at a real book.
• Whether the state file is safe under concurrency. src/state.save() replaces atomically, which is correct for one writer and is not a concurrency model; two schedulers advancing the same account have never been run and would race. On a RUNNING TOTAL a lost cycle is a permanently wrong number rather than a missing row.
• Whether a truncated reply can ever be partially trusted. The stateless control lost one call to the ceiling and src/monitor._parse rejected it outright, which is the safe behaviour and is not the same as having measured how often a fragment would have been right.
• Whether the grep-shaped no-write-off assertion would catch a write-off path written under a name it does not know. It asserts the absence of four function names and two method calls; a path called something else would pass it.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

No forced release and no write-off, non-configurable. This kit produces a proximity band, a clock status, a day count and a blocker code for a person to read. It never issues, releases, adjusts, suppresses or writes off a charge, and there is no setting that makes it. Separately: the carried accrual is written by code from the parsed dates and the previous accrual; the model's four answers are scored and are never written back into the history the next cycle is judged against.

Everywhere and nowhere -- it is a property of what is ABSENT. The only writers in the kit are evals/run.py (results/*.json) and src/state.save (data/state.json). src/state.advance() -> src/backbill.step() is the only thing that touches the accrual, and it is called by evals/run.py after every cycle including one whose call FAILED.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• IT IS NOT A CHECK ON WHETHER THE ANSWER IS RIGHT. The accrual is correct whatever the model says, which means a wrong band is reported wrongly for that cycle and nothing catches it -- see Eval.taxonomy's single band miss, which this guardrail does nothing to prevent.
• IT IS NOT AN APPROVAL GATE, BECAUSE THERE IS NOTHING TO APPROVE. There is no apply path to gate: nothing here releases or writes off anything, so there is no 'undo' because there is nothing that could 'do'. A deployment that adds one adds the gate at the same time or it has no guardrail at all.
• ⚠︎ IT IS NOT A GUARANTEE ABOUT THE WINDOW ITSELF. The back-billing window and every lead time in src/backbill.py are invented. A watchlist that is arithmetically perfect against an invented rule is arithmetically perfect against nothing.
• It does not make the state file safe to share. One writer, one file, replaced atomically -- two schedulers on one account would race, and on a running total the loser's cycle is a permanently wrong number rather than a missing row.
• It does not validate the blocker taxonomy. Six codes, invented, and the run's worst field. One of the six is arguably named for something other than what it covers -- see Eval.could_not_verify.

WatchedWhat is watched, and why that one

The kit measures 33 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

11 measured by the latest run 22 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (cycles_scored 150, watchlist_cells 54) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

3 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 3 runs measured the same system — they differ on documents, max_tokens, provider, stateless, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 3 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-run evals/check_labels.py (free) on any change to tools/build_corpus.py, src/backbill.py, src/segment.py or src/select.py -- it refuses to let a run spend if the corpus, the answer key, the experimental control, the privacy guard or the no-write-off property has moved. Re-run the scored eval AND its stateless control together (paid, 300 calls) on any change to src/prompt.py or src/state.py: the headline is a difference, so one arm re-run alone is not comparable with the other's old figure. evals/ambiguity.py is free and should be re-run whenever the blocker taxonomy is touched.

In one lineNo framework, deliberately

No framework and no dependencies at all -- json, os, re, random, datetime, argparse, urllib, http.server and concurrent.futures from the standard library. requirements.txt names nothing and says in a comment that the emptiness is load-bearing: a line there means something under src/ or evals/ imports it.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph, and the one place it looks like one is a straight line. Each account is a chain of three steps -- cycle 1, cycle 2, cycle 3 -- with no branching and exactly one edge between consecutive cycles, carrying four fields. Different accounts never touch. A framework would add an orchestrator to a for-loop that already runs 50 chains wide.

The other sideWhat a framework costs you

• No scheduler. This is a monitor that carries state and is still INVOKED, not woken -- the cadence, the retry policy and the question of what to do when a cycle does not arrive are all outside the kit.
• No state store worth the name. data/state.json is one file replaced atomically; a framework would bring a checkpointer with a concurrency model, which this does not have -- and on a running total that gap is worse than on a per-request kit.
• No observability beyond what evals/run.py prints and writes to results/. There is no tracing and no dashboard integration.
• No retry/backoff beyond src/adapters' own bounded retry, which covers a busy provider and a dropped connection and nothing else.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-unbilled-watch on the same tier, memory removed (THE CONTROL), 2026-08-23. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 2 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

1 run not plotted. b000-unbilled-watch-onepage recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — one period report goes whole into one call, and the prior period's carried state goes with it.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved; every account, tariff, meter, reading, exception entry, household name and address is invented here. Verified against the repository's own LICENSE file on 2026-08-23. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe band, the clock, the day count and the blocker code, per cycle, exact match against the computed answer key

For each of the 150 cycles and each of the four answered fields, did the reply equal the computed answer key? Band, clock and blocker are compared exactly; the day count is compared as a whole number, and a reply that is not a whole number of days is a MISS rather than a zero -- 0 is a meaningful answer here (it is what an unprotected account gets) and a parse failure must not be scored as one.

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model. The same function evals/baseline.py and evals/ambiguity.py are scored through.

Every grader on these pages scored the same 150 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 150 per field. A cycle whose reply did not parse counts as a MISS in every field, never as an exclusion.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, computed by src/backbill.replay over the planted inputs at generation time. This grader IS the reference, so its own TPR and TNR are not measured -- scoring it against itself would be circular.

How tight can the band be? No tuned threshold anywhere on the model path -- the reply decides. The denominators are printed beside every rate because two of them are small: 54 watchlist cycles and 25 memory-dependent cycles, so one row moves the first by 1.9 points and the second by 4.0.

Cadence: Re-run evals/check_labels.py (free) on any change to the corpus generator, src/backbill.py, src/segment.py or src/select.py. Re-run the scored eval (paid, one call per cycle) on any change to src/prompt.py or src/state.py -- both of those change what the model is told.

The decisionWhen to reach for it

In one lineBand accuracy restricted to the cycles that need somebody to touch them

Of the 54 cycles whose correct band is not WITHIN_LIMIT and not NO_LIMIT, how many were banded correctly? This is the number the kit is actually about; the 150-cycle average is 64 pct quiet and something that answers WITHIN_LIMIT to everything scores well on it.

MethodHow the test was run

evals/scoring.py::score, the same pass as the reference grader -- a slice of the same cells, not a second comparison.

Every grader on these pages scored the same 150 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

hits among band cells where gold is APPROACHING, AT_RISK or BREACHED, divided by 54.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the same data/gold.jsonl the reference grader uses, restricted to cells whose band is not WITHIN_LIMIT or NO_LIMIT. It measures the reference grader's own answers on a subset, so it has no independent standard.

How tight can the band be? 54 rows means the finest honest band is about 1.9 points. Nothing is tuned.

Cadence: Whenever the reference grader runs -- it is the same pass.

The decisionWhen to reach for it

In one lineFalse alarms on the cycles where nothing needs doing

Of the 96 cycles whose correct band is WITHIN_LIMIT or NO_LIMIT, how many were put on the watchlist anyway? This is the other direction, and it costs somebody an hour where the first costs the balance.

MethodHow the test was run

evals/scoring.py::score, the same pass as the reference grader.

Every grader on these pages scored the same 150 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

quiet cells whose reply names a watchlist band, divided by 96.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the same data/gold.jsonl, restricted to the 96 cells whose band is WITHIN_LIMIT or NO_LIMIT.

How tight can the band be? 96 rows, so one row is 1.04 points. No threshold is tuned.

Cadence: Whenever the reference grader runs -- it is the same pass.

The decisionWhen to reach for it

In one lineThe cycles whose answer cannot be derived from their own page

On the 25 cycles where a suspension began, or a dispute was raised, in a statement the model is not holding, was the band right and was the day count right? This is the subset the whole monitor argument rests on, and it is marked in the answer key by the generator's own structure -- never by what any arm answered.

MethodHow the test was run

evals/scoring.py::score, the same pass as the reference grader.

Every grader on these pages scored the same 150 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

hits among cells where gold.memory_dependent is true, divided by 25, reported separately for the band and for the day count.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the same data/gold.jsonl, restricted to the 25 cells the generator marked memory_dependent. evals/check_labels.py asserts that count and asserts that none of those cells is answerable from its own page.

How tight can the band be? 25 rows. One row is 4.0 points, so no band finer than about four points means anything on this grader.

Cadence: Whenever the reference grader runs -- it is the same pass.

The decisionWhen to reach for it

In one lineThe portfolio total reported as beyond the back-billing window

Sum the unbilled value of every cycle the arm reported BREACHED, and compare it with the truth. This is the number that lands in a month-end pack: how much of the unbilled book is already unrecoverable. It is a DIFFERENT question from per-account accuracy and it is the question the catalogue row asks -- 'reconcile that against this month's unbilled-revenue estimate'.

MethodHow the test was run

evals/scoring.py::score, the same pass as the reference grader.

Every grader on these pages scored the same 150 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

abs(sum(value where reported band == BREACHED) - sum(value where gold band == BREACHED)) / the gold total. The gold total is $29,024.35 of $158,520.97 unbilled in the whole book.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the unbilled_value_usd field of data/gold.jsonl, summed over cells whose gold band is BREACHED. The values themselves are generated and asserted against a per-fuel household range by evals/check_labels.py.

How tight can the band be? No threshold. The gold total is dominated by 18 cycles across 6 accounts, so one account moves it by several percent -- the denominator is small in accounts even though it is large in dollars.

Cadence: Whenever the reference grader runs -- it is the same pass.

The decisionWhen to reach for it