The business caseThe problem this solves

Deciding why an invoice line's charged quantity differs from the usage behind it is arithmetic with a priority order — which blocks come out of the mediated total, which one deliberately stays in, what the billing increment is for THIS service, and only then what the remaining gap can be blamed on. The record puts four louder things next to that arithmetic: a duplicate-suspect figure that review already cleared, a late-arrival figure that belongs to last month's invoice, a suspense bucket that is still owed, and the billing analyst's own note, which is the one part of the record that is nobody's measurement. Someone opening each invoice line beside its mediated-usage summary, subtracting the previous period's late arrivals and the CONFIRMED duplicates (not the larger suspect figure beside them), leaving the suspense bucket IN because unrated usage is still owed, rounding what is left up to that service's own billing increment, and only then deciding what the remaining gap is. It is a minute per line and it is every line, and the part that goes wrong is not the subtraction: it is reaching for the loudest number on the page — the duplicate suspects, the late-arrival figure, the heavy suspense bucket — each of which produces a confident, well-formed, wrong answer.

The inputThe actual reconciliation records

The corpus is 55 reconciliation records, 0.03 MB (txt 55). Plain text, one format, invented rather than fetched — a real usage-to-invoice reconciliation record carries a named customer's line, the call-detail traffic behind it and a billing analyst's own note about that specific account, and there is no public corpus of (mediated usage, invoiced quantity, correct cause) triples for the same reason there is no public corpus of bank statements. Generating it also makes the label mechanical: gold is the arithmetic, not somebody's reading of the note.

Invoice Line
------------
TL-JH-61326

Rating Domain
-------------
MED-CALDERA-03

Service Type
------------
data

Billing Period
--------------
2026-11

Mediated Usage
--------------
535715 KB

Invoiced Quantity
-----------------
536576 KB

Unrated Usage
-------------
0 KB

Prior Period Usage
------------------
0 KB

Duplicate Suspects
------------------
23714 KB

Confirmed Duplicates
--------------------
18595 KB

Invoice Status
--------------
issued

Analyst Note
------------
Duplicate session records on this line; it looks double-charged to me.

The outcomeWhat a good result looks like

An eleven-field extracted record per invoice line, plus one pure-code routing decision taken from two of those fields: a line that OVER-billed the customer on an invoice already issued is the one that goes to billing today. Nothing here issues a credit, reverses a charge, re-rates a suspense bucket or contacts a customer.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Screening a billing cycle for lines with an actionable variance, before a revenue-assurance analyst opens them — either tier on quality — they are identical on every line both reached
  100 pct six-way cause accuracy on the fast tier over all 55 records, against the free analyst-note floor's 60.0 pct (22 wrong, including 3 real variances called clean and 9 false alarms).
• Deciding the two tiers on cost, speed or accuracy — the fast tier
  About 5 pct cheaper per record ($0.0026305 vs $0.0027734), 32 pct lower p50 latency (4824 ms vs 7125 ms), and identical on every published grader over every record both reached. It also completed all 55; the deliberating tier's run completed 54.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-usage-variance. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts, the 55-record picker populates and the field table draws its eleven empty rows; clicking Extract returns the no-key sentence rather than a stack trace. python -m evals.check_labels passes with no network access at all.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per reconciliation record

Current processWhat it replaces

Someone opening each invoice line beside its mediated-usage summary, subtracting the previous period's late arrivals and the CONFIRMED duplicates (not the larger suspect figure beside them), leaving the suspense bucket IN because unrated usage is still owed, rounding what is left up to that service's own billing increment, and only then deciding what the remaining gap is. It is a minute per line and it is every line, and the part that goes wrong is not the subtraction: it is reaching for the loudest number on the page — the duplicate suspects, the late-arrival figure, the heavy suspense bucket — each of which produces a confident, well-formed, wrong answer.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 0 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings; a record with none falls back to one whole-document segment

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 55 records cut into 660 sections in under three thousandths of a second, in process, with no model and no network.

LicenceLicence

MIT — this repository's own licence. Every line id, rating-domain name, quantity and analyst note is invented; no real carrier, customer, filed tariff or interconnect agreement is named or reproduced.

Bring your ownBring your own reconciliation records

Replace data/corpus/*.txt, write your own data/fields.json, and supply a gold record per line. SECTION_HINTS in src/select.py maps fields to section headings and will need editing for a different reconciliation-report layout; when it does not match, selection falls back to the whole document — slower, more expensive, always correct. increment(), classify() and compute() in src/extract.py are this kit's own invented increment table, variance rule and credit rule, and should be the first things you replace with your own rating configuration and credit policy.

⚠︎ And what stops being true when you do: Every score on this page was measured on quantities that were already aggregated correctly. Point this at your own mediation output and the aggregation becomes part of the system under test, and nothing measured here transfers to it.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 1,799 tokens assembled, 880 are instructions — 49% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on TLV-0052 — three calls at max_tokens=1, each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. The total (1799) matches the input_tokens the worked example's own call reported, which is what proves the measured decomposition is of the prompt actually sent. kits/UC0040-usage-variance/results/tokens-p001-usage-variance.json.

Full promptThe prompt in full

You extract structured fields from a telecom usage-to-invoice reconciliation record for one invoice line. You return JSON and nothing else.

RULES, in order of importance:
1. If the record does not state a field, return null for it. Do not infer it and do not use what you know about the world. A field stated as 0 is stated -- return 0, not null.
2. `variance_cause` is decided by ARITHMETIC over five structured quantities -- mediated_quantity, invoiced_quantity, unrated_quantity, prior_period_quantity and confirmed_duplicate_quantity -- never by how the analyst note reads and never from the Duplicate Suspects figure. Work it out yourself, in this order:
   a. THE BILLING INCREMENT COMES FROM service_type. voice is billed in whole minutes, so the increment is 60 seconds. data is billed in whole megabytes, so the increment is 1024 KB. sms is billed per message, so the increment is 1 and there is NO rounding tolerance at all.
   b. billable = mediated_quantity - prior_period_quantity - confirmed_duplicate_quantity. DO NOT SUBTRACT unrated_quantity: usage that failed rating is still this period's usage and is still owed.
   c. expected = billable rounded UP to the next whole multiple of the increment.
   d. gap = invoiced_quantity - expected.
   e. Classify the gap and STOP AT THE FIRST MATCH:
      - gap is exactly 0 -> 'none'
      - the absolute value of gap is LESS THAN the increment -> 'rounding'
      - gap is negative, unrated_quantity is above 0, and the size of gap is within one increment of unrated_quantity -> 'unrated_usage'
      - gap is positive, confirmed_duplicate_quantity is above 0, and gap is within one increment of confirmed_duplicate_quantity -> 'duplicate_records'
      - gap is positive, prior_period_quantity is above 0, and gap is within one increment of prior_period_quantity -> 'late_records'
      - otherwise -> 'unexplained'
3. THE ROUNDING CHECK COMES BEFORE ANY CAUSE IS NAMED. A gap smaller than one increment is explained by rounding and nothing else -- do not name a cause for it, even when one of the three blocks happens to be about that size.
4. CONFIRMED DUPLICATES ARE NOT DUPLICATE SUSPECTS. The record states both. De-duplication FLAGS suspects; review CONFIRMS a subset of them, and a suspect that review cleared is two genuinely distinct sessions. Only the Confirmed Duplicates figure enters the arithmetic, and confirmed_duplicate_quantity is read from that section. The suspect figure is usually the larger of the two and is often zero once confirmed.
5. THE PRESENCE OF PRIOR-PERIOD USAGE IS NOT ITSELF A VARIANCE. Records that arrived after the collection cutoff belong to the PREVIOUS invoice. Subtract them, and if the invoice already left them out the answer is 'none'.
6. THE ANALYST NOTE IS A FIELD TO COPY, NOT EVIDENCE ABOUT THE CAUSE. A note naming duplicate records, unrated usage, late records or rounding does NOT make that the cause, and a note saying the line reconciled cleanly does NOT mean it did. The five quantities decide; the note is the analyst's own remark and may disagree with them.
7. Copy values verbatim from the record wherever possible, and report every quantity as a bare number with the unit left out of it.
8. Use the exact allowed value for a field that lists them.
9. Return every field named in the schema, even when the answer is null.

Extract these fields:
- line_id (string) -- the invoice line identifier, verbatim
- service_type (enum) one of: voice, data, sms -- which service this invoice line bills, verbatim
- billing_period (string) -- the billing period, YYYY-MM
- mediated_quantity (number) -- the total usage mediation produced for this line and period, as a bare number without the unit
- invoiced_quantity (number) -- the quantity the invoice line actually charged, as a bare number without the unit
- unrated_quantity (number) -- usage that failed rating and is sitting in the suspense bucket, as a bare number without the unit. It is part of the mediated total. It may be 0
- prior_period_quantity (number) -- usage inside the mediated total that arrived after the collection cutoff and belongs to the PREVIOUS billing period, as a bare number without the unit. It may be 0
- confirmed_duplicate_quantity (number) -- usage inside the mediated total that de-duplication review CONFIRMED as duplicate records, as a bare number without the unit. Read the Confirmed Duplicates section, NOT the Duplicate Suspects section -- a suspect that review cleared is two genuinely distinct sessions and is not a duplicate. It may be 0
- invoice_status (enum) one of: draft, issued -- has this invoice already been issued to the customer, or is it still a draft?
- analyst_note (string) -- the billing analyst's own free-text note on the line, copied verbatim
- variance_cause (enum) one of: none, rounding, unrated_usage, duplicate_records, late_records, unexplained -- why the invoiced quantity differs from the usage that should have been billed. Decide this STRICTLY by arithmetic over mediated_quantity, invoiced_quantity, unrated_quantity, prior_period_quantity and confirmed_duplicate_quantity -- never from analyst_note and never from the Duplicate Suspects figure. Step 1, the billing increment for this service: voice is billed in whole minutes, so the increment is 60 seconds; data is billed in whole megabytes, so the increment is 1024 KB; SMS is billed per message, so the increment is 1 and there is NO rounding tolerance at all. Step 2, billable = mediated_quantity - prior_period_quantity - confirmed_duplicate_quantity. Unrated usage is NOT subtracted: it failed rating but it is still this period's usage and is still owed. Step 3, expected = billable rounded UP to the next whole increment. Step 4, gap = invoiced_quantity - expected. Step 5, classify the gap in this order and stop at the first match: if gap is 0 answer 'none'; else if the absolute value of gap is smaller than the increment answer 'rounding'; else if gap is negative and its size is within one increment of unrated_quantity (which must be above 0) answer 'unrated_usage'; else if gap is positive and within one increment of confirmed_duplicate_quantity (which must be above 0) answer 'duplicate_records'; else if gap is positive and within one increment of prior_period_quantity (which must be above 0) answer 'late_records'; otherwise answer 'unexplained'.

Return a JSON object with exactly these keys: line_id, service_type, billing_period, mediated_quantity, invoiced_quantity, unrated_quantity, prior_period_quantity, confirmed_duplicate_quantity, invoice_status, analyst_note, variance_cause
Use null for any field the record does not state.

USAGE-TO-INVOICE RECONCILIATION RECORD
--------------------------------------
Invoice Line
------------
TL-RA-64456

Service Type
------------
data

Billing Period
--------------
2027-05

Mediated Usage
--------------
1170495 KB

Invoiced Quantity
-----------------
1143808 KB

Unrated Usage
-------------
29496 KB

Prior Period Usage
------------------
26783 KB

Duplicate Suspects
------------------
28520 KB

Confirmed Duplicates
--------------------
22632 KB

Invoice Status
--------------
issued

Analyst Note
------------
Traffic landed after the collection cutoff; I think last month's usage slipped onto this bill.

Raw responseThe raw response

{"line_id":"TL-RA-64456","service_type":"data","billing_period":"2027-05","mediated_quantity":1170495,"invoiced_quantity":1143808,"unrated_quantity":29496,"prior_period_quantity":26783,"confirmed_duplicate_quantity":22632,"invoice_status":"issued","analyst_note":"Traffic landed after the collection cutoff; I think last month's usage slipped onto this bill.","variance_cause":"duplicate_records"}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check a telecom invoice line against its usage — 55 reconciliation records. Two tiers of one model family answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran r001 and r002 publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersThree ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The analyst-note floor (evals/baseline.py) and the scorer (evals/judge.py) are both pure code and cost $0.00 to run against either result set. The figure above is both scored runs' own token counts (55 records on the fast tier, 54 on the deliberating tier) priced at Google Gemini 3 Flash's published rate — the same basis cost_per_query_usd uses, not a second, larger spend. It excludes the eight calls spent outside the scored runs: one MAX_TOKENS calibration, a two-call wiring probe, three prompt-token measurements, one worked-example capture and one live screenshot.

Cost driversWhat actually moves the bill

• THE REASONING PASS NOBODY ASKED FOR. The calibration call billed 552 output tokens for 381 characters of JSON, and the provider's own breakdown says 444 of them (80 pct) were reasoning tokens that never reach the text; the worked example repeated it at 390 of 502 (78 pct). Output is priced six times input on the published card, so this is the largest single line on this kit's bill — and nothing in the kit requested it. src/adapters/__init__.py sends a thinking parameter only when a caller passes one and this kit's harness never does, so it is the provider's own default and every recorded run paid for it.
• The fixed system prompt and field schema (1588 of 1799 tokens on the example call, 88 pct) outweigh the record sections sent (211 tokens) — the floor every call pays before a single quantity is read. Most of that floor is the RULE: the variance-cause hint alone is the longest line in data/fields.json, because the whole priority order is stated rather than inferred.
• Output length: the model returns a full eleven-key JSON record every call, including the billing analyst's note copied back verbatim, whatever the record says.

Your volumeWhat it costs at your volume

Linear in invoice lines: each call is independent, carries the same fixed prompt and shares nothing with its neighbours, so 550 lines cost ten times 55 and take ten times as long. Nothing amortises — there is no index to build and no cache — which is also why there is no volume discount to find without changing the design. What does NOT scale linearly is the failure count: one transport loss in 110 calls is one line at this size and thousands at a carrier's.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 55 usage-to-invoice reconciliation records, one completion call each, on the fast tier. The deliberating tier's own 54 completed calls are recorded separately in Cost.cost_by_model and are not projected here.

The codeEvery file, and what it is for

Five modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the reconciliation record into addressable sections, pure code

# Cut a document into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections carry each field, pure code — the Rating Domain section is mapped by nothing and never sent, while Duplicate Suspects is mapped IN on purpose, because a kit that hid the decoy would have solved the problem for the model

You change it to: map fields to your own reconciliation report's headings; unmatched falls back to the whole document

# Pick which sections plausibly carry each field. Pure code -- the last deterministic step

SECTION_HINTS = {
def for_field(secs, field):
def plan(secs, fields):

src/prompt.pyprompt

assemble one call for all eleven fields, with the whole rule stated — the per-service billing increment, which blocks leave the mediated total, that unrated usage stays in, and that rounding is checked before any cause is named

# Assemble the extraction prompt. One prompt per usage-to-invoice reconciliation record, all

SYSTEM = (
def field_schema(fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key — plus the pure-code business-condition check downstream: an over-billed line on an invoice already issued is routed for a credit

You change it to: the credit rule — this kit ships "over-billed AND already issued", and a real desk weighs the dollar size, the dispute window and any contracted tolerance. It is deliberately NOT the same function as classify(), so changing WHO GETS A CREDIT does not change WHAT THE VARIANCE IS

# Extract one usage-to-invoice reconciliation record's fields: segment, select, prompt, one model

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 3000
MEASURED_ON = ("calib-c001-usage-variance, TLV-0005, 552 output tokens (444 of them reasoning), "
CAUSES = ("none", "rounding", "unrated_usage", "duplicate_records", "late_records", "unexplained")
def load_fields():
def load_doc(line_ref):
def documents():

evals/judge.pyjudge

score field accuracy, the six-way cause grade, the actionable/not confusion matrix and the credit flag separately, pure code

# Score an extraction run. PURE CODE -- gold is exact and the answer is one value per cell, so

ACTIONABLE = ("unrated_usage", "duplicate_records", "late_records", "unexplained")
def norm(v):
def _num(v):
def equal(field, got, want):
def score(fields, records, golds):
def _matrix(rows, positive):
def score_flags(records, flags, golds):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 1792 input and 578 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's reconciliation records are entirely synthetic (tools/build_corpus.py, seed 20260822): no real carrier, customer, line or mediation platform exists in the corpus, and nothing was fetched from anywhere. The only outbound traffic the kit makes is one chat-completion request per record to the configured provider, carrying the mapped sections of one reconciliation record. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env, this kit's own .env (which pins only MODEL) or the real environment, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser — and that path was exercised for real here, because r002's transport error surfaced through it.

The experimentWe did not attack it — and three of four boundaries hold

Two of the three boundaries that hold were confirmed by reading the code: no code path raises a credit or contacts anybody, and the routing rule reads only enums and integers, so the analyst's note cannot reach it. The third was confirmed by reading the code AND then observed live — r002's TLS handshake failure travelled through src/app.py's redaction path and reached the result file with no host and no credential in it. The fourth is open. An indirect prompt injection needs a field an outside party authored, and this kit has exactly one, which makes it the obvious place to attack and the reason the gap is named rather than glossed. Confirmed by reading the code, not by a run, on 2026-08-22 -- no attack run was fired; see redteam.why.

The first three boundaries hold — two confirmed by reading the code, and the third also observed live when a real transport error travelled through the redaction path. The fourth is open and is named as open: an indirect prompt injection in the analyst's note is exactly the attack this kit's shape invites, and nothing here has tried it.

The result0 attack trials, and three of four boundaries checked here hold on evidence. The one that does not is the one this kit's own shape invites: the billing analyst's note is free text an outside party wrote, and whether an instruction hidden in it could move the variance cause is unmeasured.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted ambiguity is a REGISTER mismatch plus three NUMERIC decoys — a note naming the wrong cause, a duplicate-suspect figure review already cleared, a late-arrival figure belonging to last month, and a suspense bucket that is still owed. All four measure whether a model does the arithmetic when something louder points elsewhere. None of them measures whether a model obeys an instruction hidden in the same field.

HonestyWhat this does not prove

• Whether an indirect prompt injection in the analyst_note field could move variance_cause on either tier. No attack run exists.
• Whether the kit behaves safely against a hostile provider — a response body crafted to break the JSON extraction in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed document, and nothing beyond that was tested.
• Whether a real carrier's reconciliation archive would carry anything sensitive this kit mishandles. The corpus has no personal data by construction; a real record identifies a subscriber line and the traffic behind it, and none of that path is exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_credit fires when the variance OVER-billed the customer AND the invoice has already been issued — variance_cause in ("duplicate_records", "late_records") and invoice_status == "issued". `unrated_usage` is UNDER-billing and deliberately does not fire it: real revenue leakage, but nobody is owed money. Both values come out of the same reply; the rule is run afterwards in pure code, over whatever the model returned, and never over gold. A reply missing either value returns None rather than False: an unknown is not a pass.

src/extract.py::compute(), called by extract() on every record, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by identical code. The variance arithmetic it reads lives in SEPARATE functions — increment(), expected_invoiced() and classify() — which are also what the corpus generator wrote gold with and what src/prompt.py states to the model. One definition, three readers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a check on whether the extracted quantities are right. If the model misreads the mediated total and then classifies that misreading consistently, the line is routed (or not routed) on a wrong number and this rule cannot tell. The consistency diagnostic in evals/judge.py is the closest thing to that check, it is reported separately, and it is itself blind to the same case.
• It is NOT a real carrier's billing-adjustment policy. Over-billed-and-issued is this kit's own simplification, invented for this corpus. No published tariff, settlement rule, regulatory requirement or operator's own credit procedure was consulted, and none is reproduced. A real billing desk weighs the dollar size of the credit, the dispute window, and whether the line sits inside a contracted tolerance.
• It is NOT a disposition. Nothing here raises a credit, reverses a charge or re-rates a suspense bucket, and variance_cause is an extracted field rather than a decision anybody is bound by.

WatchedWhat is watched, and why that one

The kit measures 16 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 9 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

3 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 3 runs measured the same system — they differ on documents, extraction_cells, failures, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 3 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute(), classify() or increment() in src/extract.py, on any change to data/fields.json's allowed values for variance_cause or invoice_status, and on any corpus regeneration. Changing WHO GETS A CREDIT is a policy change and must be re-scored, even though it never changes what the variance is.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per record — segment, select, prompt, one call, parse, classify, compute — with no branch, no loop and no agent. Anything that looks like orchestration in a kit this size is a diagram of a straight line.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain: the JSON extraction from a fenced reply, the retry/backoff policy, the section regex and the .env reader are all code somebody has to own. This kit found a real bug in one of them the hard way — see the transport-retry note above.
• No framework means no framework's ecosystem — no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. What ships is what is in the repository.
• The provider abstraction covers exactly two wire formats. A third provider is one function and one dict entry, and until somebody writes it the kit runs on two shapes.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-usage-variance on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 2 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

1 run not plotted. b000-usage-variance-rules recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Every line id, rating-domain name, quantity and analyst note is invented; no real carrier, customer, filed tariff or interconnect agreement is named or reproduced. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-field exact match, light normalisation

Does the model's value for each field match gold, after trimming whitespace and punctuation and treating numbers within half a cent as equal? Nothing in this corpus is legitimately null — every quantity is stated on every record INCLUDING the zeros — so a null is always a miss here, and null and 0 are scored as different answers to "how much failed rating", because only one of them is a measurement.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every call, and the same field-match logic the free baseline is scored by — a baseline and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same five quantities, service type and invoice status the document states; evals/check_labels.py asserts every field is populated on every row, that the three blocks fit inside the mediated total, that every cause label agrees with its own arithmetic, that no row matches two cause branches, and that the rounding boundary holds at the edge, before any run is allowed to spend.

How tight can the band be? There is no tolerance band on the field grade itself — it is exact match after trimming whitespace and punctuation, and numbers within half a cent are treated as equal, never a continuous score to round.

Cadence: Re-score on any change to src/prompt.py, data/fields.json or tools/build_corpus.py — the first two change what is asked, the third changes what is asked about. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one linevariance_cause against gold's own arithmetic

Did the run name the RIGHT cause, out of six? And, collapsed onto the only binary a desk queues on, did it spot that there was an actionable variance at all — anything other than none or rounding? The two readings are reported side by side and must never be averaged: a reply that says duplicate_records where the truth is late_records has seen the variance and blamed the wrong block, which is a different day's work from missing it.

MethodHow the test was run

evals/judge.py::score_flags, in-process, no key and no model.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's cause is re-derived inside the grader by the same arithmetic the kit publishes — the per-service increment, the two blocks that leave the total, the one that stays in, rounding checked before any cause is named — so the truth this grades against can never be a separately-typed label that drifted from the rule.

How tight can the band be? No threshold — the cause is one of six allowed values, and a reply that returns none of them is counted as unanswered rather than folded into a correct cell.

Cadence: Re-run whenever classify() or increment() in src/extract.py changes, whenever the corpus is regenerated, and on any provider or model change.

The decisionWhen to reach for it

In one lineneeds_credit against the same rule run over gold

Does the pure-code routing decision — over-billed AND already issued — land on the same lines it would land on if both fields had been read perfectly? It is a BUSINESS CONDITION, so unlike a self-consistency check it genuinely needs labels, and saying so is half of what makes the number believable.

MethodHow the test was run

evals/judge.py::score_flags, in-process, no key and no model.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute(), the same function the run uses, applied to GOLD's variance_cause and invoice_status. One rule, two inputs, so a change to the rule moves both sides together and the grader cannot silently grade an old policy.

How tight can the band be? No threshold — one enum test and one boolean, ANDed.

Cadence: Re-run whenever compute() changes. Changing WHO GETS A CREDIT is a policy change and must be re-scored, even though it never changes what the variance is.

The decisionWhen to reach for it