The business caseThe problem this solves

"Which completed jobs have no invoice line" is a SELECT and nobody needs a language model for it. The question a billing operation actually has to answer is the next one: does the invoice register genuinely BILL this job -- by its exact number, a bundled route-week line, or a reformatted job reference -- or does a line merely QUOTE the job's number while reversing the charge? Is a note that sounds like a billing hold an actual, dated, formal one, or a customer's request that must not stop the clock? And does the job even have the evidence on file to be billed at all? Three of those four questions can sit in a register line or a desk note that a foreign-key join never reads. Someone manually reconciling completed work orders against the invoice register: pulling the week's register, checking whether each job's exact reference appears anywhere in it, opening any line that looks like a bundled or corrected charge to work out what it actually covers, reading exception notes to decide whether a job is genuinely on hold or just has a customer complaint attached, and tracking which weight-based jobs are still missing a scale ticket.

The inputThe actual reconciliation snapshots

The corpus is 150 reconciliation snapshots, 0.85 MB (json 1 · jsonl 1 · txt 150). AN UNBILLED-WORK RECONCILIATION QUEUE JOINS OPERATIONS TO REVENUE, which makes it one of the more commercially sensitive files a hauler holds -- it names customers, exactly what was done at each site, and which completed jobs the billing desk has not yet been able to invoice. There is no public one and there never will be. Generating it also bought the one thing a captured corpus cannot give: the answer key is src/billing.step's output over the planted per-run facts, so a whole-window hold convention this fiddly cannot carry its author's misreading into the score. ⚠︎ AND THE RULE IS INVENTED. The nine rules, the 21-day supervisor-review threshold and the whole-window day-counting convention reproduce no billing SLA, no revenue-assurance policy, no waste-hauling regulator's guidance and no real hauler's escalation practice, and they name none. ⚑ THIS PACK IS DELIBERATELY, ONLY OPERATIONAL. It answers 'is this job billed, and if not, how long and why' from the completion date and the invoice register alone -- it does not read, gate on, or feed a revenue-recognition close calendar or a GL posting period, and nothing here claims those two clocks agree. Whether unbilled-candidate detection should interface with accounting-close timing is an open question for whoever owns that boundary, left unresolved here on purpose (default: stay operational) rather than guessed at.

Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated unbilled-work reconciliation
snapshot for an AI use-case kit; it reproduces no real hauler, customer, contract or
billing register. The billing rule and its threshold are ILLUSTRATIVE.

Work Order
----------------------------------------------------------------
Customer                       : CUST-8021 (Grandview Retail Plaza)
Work order reference           : WO-0001
Service type                   : COMPACTOR_HAUL
Container / equipment          : CMP-6207
Route                          : RT-04
Completion date                : 2026-02-21
Contract value for this job    : 1,766.72 USD
Evidence required for billing  : driver completion confirmation AND a weight ticket

Billing Rule
----------------------------------------------------------------
Supervisor review threshold    : 21 days unbilled (operator-tunable; see .env
                                SUPERVISOR_REVIEW_DAYS)

Rule B-1  A completed work order becomes a BILLING CANDIDATE the day after its completion date,
          and stays one until an invoice line in the register genuinely references it.
Rule B-2  Days unbilled are counted in whole RECONCILIATION WINDOWS. The first window runs from the
          completion date to the first reconciliation run; every window after that is exactly the
          7-day cadence. A window in which a formal billing hold (Rule B-3) was open for ANY part
          does not count toward the total AT ALL; a window with no formal hold open at any point
          counts in full. This is a stated simplification -- a hold that closes partway through a
          window still excludes the whole window -- and it is not split into partial days.

The outcomeWhat a good result looks like

Every completed work order carries a status (BILLED, UNBILLED, ON_HOLD, REVIEW or CONTEXT_INCOMPLETE), a hold-adjusted day count, an evidence-completeness flag, and a routing recommendation -- NONE for a billed job, ANALYST for a routine or evidence-incomplete one, SUPERVISOR for one that is aged past the review threshold or genuinely on hold.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• a billing desk with a clean, single-ID invoice register and no bundled or corrected lines — the exact-join-mem floor (b001)
  91.33 pct for $0.00, and it already beats the model on nothing this scenario tests
• a billing desk whose register carries bundled route-week lines or reformatted job refs, and whose desk notes carry informal hold-sounding language — the model, with the carried state
  the two traps this kit exists to test -- a bundled/reformatted reference and a hold-sounding note with no formal entry -- are exactly where every free floor in this run loses points, and the model lost none
• deciding whether ANY of this is worth a model call at all — run all three floors first (0 calls, $0.00) and look at the per-bucket cut
  b001 already solves the hold-decoy trap by SCOPE alone; the model's margin over the best floor lives entirely in bundled/reformatted matching and the reversal decoy, which is a much narrower claim than 'the model beats free code'

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-24. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-24 — r001-unbilled-work. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with NO key configured reproduces: the 150 snapshots and the answer key (regenerated from the seed in well under a second), all eighteen pre-flight assertions, all three free floors scored end to end, the wiring stub, and the local UI at 127.0.0.1:8208 including what run r001-unbilled-work recorded for every reading. What it CANNOT reproduce without a key is a model column of its own.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per reading: one HTTP request carrying the assembled prompt, and the reply parsed to a status, a day count, an evidence flag and a routing call. Splitting the snapshot into sections, dropping the withheld one, and computing the whole-window hold arithmetic all happen outside this measurement and cost no network at all. p95 (21.0s) is 3.7 times p50 (5.6s); the spread tracks how much the invoice-match decision needs reasoning about -- an exact match is a lookup, a bundled or reformatted reference or a reversal decoy is a reading task.

Current processWhat it replaces

Someone manually reconciling completed work orders against the invoice register: pulling the week's register, checking whether each job's exact reference appears anywhere in it, opening any line that looks like a bundled or corrected charge to work out what it actually covers, reading exception notes to decide whether a job is genuinely on hold or just has a customer complaint attached, and tracking which weight-based jobs are still missing a scale ticket.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• 3 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one work orders (3 weekly reconciliation runs each) is

No split, and no chunking. The unit is a WORK ORDER -- three consecutive weekly reconciliation runs processed strictly in order, because run 3's prompt contains a carried state produced by run 2. Each snapshot goes whole into one call, minus the one section no field asks for.

SetupWhat the setup figure measured

There is no index and no retrieval step -- the open population is re-read whole on each reconciliation run. tools/build_corpus.py writes 150 documents and the answer key from a fixed seed in well under a second with no clock read and no model called.

LicenceLicence

MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved; every customer, work order, container, invoice line, evidence-log entry and desk note is invented here. Verified against the repository's own LICENSE file on 2026-08-24.

Bring your ownBring your own reconciliation snapshots

Replace src/billing.py FIRST, and not as an option -- SUPERVISOR_REVIEW_DAYS, the whole-window hold convention and the invoice-match test are all invented here, and data/gold.jsonl is literally their output. Then point tools/build_corpus.py at your own work orders and invoice register, or replace data/corpus/*.txt and data/gold.jsonl directly. Keep the eight section headings in src/segment.py::SECTIONS -- the parser asserts all eight in every document before a run may spend -- and keep the Billing Rule block, because the rule the model applies is READ OFF THE PAGE rather than baked into the prompt. gold.jsonl needs one row per snapshot carrying the raw src/billing.step inputs (days since completion, cumulative hold days, whether the register matches, whether a hold is open, whether evidence is complete) alongside the resolved answer; evals/check_labels.py re-derives the answer from src/billing.step and refuses if they disagree.

⚠︎ And what stops being true when you do: The measured figures on this page do not travel with your register. Every one of them is a fact about FIVE curated register-line phrasings and a 21-day threshold nobody has confirmed, over a population that does not change between runs and where no work order goes on hold twice. Re-run all four arms (model, stateless control, and the three floors) on your own register's actual reference formats before quoting anything here against it.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 10,422 characters assembled, 4,553 are instructions — 44% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

TWO MESSAGES, JOINED IN SEND ORDER. The provider receives a system message and a user message; prompt_verbatim prints both, separated by a blank line, because publishing only the second would be publishing most of a prompt. Replayed from the kit's own src/prompt.build() for WO-0001-R1 with the carried state that reading was actually given (src/prompt.build_parts() produces the identical decomposition used here). The section list it produces is the list the run recorded in sections_used -- Synthetic Record, Work Order, Billing Rule, Billing Position, Invoice Register, Evidence Log, Billing Desk Notes -- and src/prompt.py is the only thing in the kit that assembles a prompt, so there is no second path a run could have taken. tokens.input (2454) is the run's own INPUT TOKEN AVERAGE over all 150 readings, not a per-reading count the harness does not store; the assembled parts above are this exact reading's, in characters.

Full promptThe prompt in full

You are a scheduled unbilled-work reconciliation watch. You apply a written billing rule to one completed work order at one reconciliation run. You answer with one JSON object and no other text.

You are a scheduled unbilled-work reconciliation watch for one waste and recycling hauler's
billing desk. It wakes every seven days, re-reads every completed work order still without a
matched invoice line, and reports each one. You are reading ONE work order at ONE reconciliation
run.

The billing rules, the work order's evidence and the invoice register excerpt are reproduced in
the snapshot below. Apply them exactly as written. You cannot see the earlier reconciliation runs;
what is known about them is stated under "Carried state" and is the only history available to you.
Do not assume anything about earlier runs beyond it.

How to decide:
 - First check the evidence (Rule B-5). A weight-based service needs a driver completion
   confirmation AND a weight ticket on file; a flat-rate service needs only the driver completion
   confirmation. "Evidence on file" on the page states this directly. If the evidence this rule
   requires is missing, the work order is CONTEXT_INCOMPLETE: days_unbilled null, evidence_complete
   NO, route_to ANALYST. Never bill or age a job whose required evidence is not on file.
 - Otherwise, decide whether the Invoice Register actually bills this job (Rule B-4). It is BILLED
   when a line genuinely references it -- by its exact work order number, by a bundled line
   listing several work order numbers for one route-week, or by a reformatted job reference. A
   line that only QUOTES this work order's number while describing a correction, a reversal or a
   charge taken back out does NOT bill it (Rule B-6) -- read what the line SAYS, not just whether
   the number appears in it. Once a work order has been reported BILLED on an earlier run it stays
   BILLED (Rule B-7); do not re-open it. IMPORTANT: "days_unbilled" is NEVER null on a BILLED
   reading -- it is a historical fact ("it took N days to bill"), not a still-running count. If the
   carried state already says BILLED, its sentence states the exact whole-day number that was
   already counted; COPY THAT NUMBER, do not recompute it and do not report null. If the register
   bills it for the FIRST time on THIS run, compute days_unbilled the same way an UNBILLED reading
   would (below) and report that number, frozen.
 - If not billed, decide whether the CURRENT reconciliation window is held (Rule B-2/B-3): either
   the Evidence Log shows a hold OPENED in this window with no matching close in it, or the carried
   state already says a hold was open coming in and this window's Evidence Log shows no CLOSE entry
   for it. A standing note that only sounds like a hold -- a customer disputing a charge, a contact
   asking to delay invoicing -- does NOT count, however it reads.
 - "Days since completion, as of this run" on the page is the total elapsed days, already computed
   for you; do not re-derive it from the calendar. The FIRST reconciliation window runs from
   completion to the first run; every window after that is exactly 7 days. Days unbilled = days
   since completion, MINUS the full length of every window (this one and any earlier one) that was
   held for any part -- a window with the hold open even briefly does not count at all (Rule B-2).
   Add that to what the carried state already established for earlier windows. Report REVIEW if the
   total is 21 or more; otherwise UNBILLED. If the current window is held, report ON_HOLD instead
   (days_unbilled may still be reported, reflecting only the non-held windows).

Answer with a single JSON object and nothing else:

{"status": "BILLED|UNBILLED|ON_HOLD|REVIEW|CONTEXT_INCOMPLETE",
 "days_unbilled": <whole days counted as unbilled, hold-adjusted; null ONLY for CONTEXT_INCOMPLETE>,
 "evidence_complete": "YES|NO",
 "route_to": "NONE|ANALYST|SUPERVISOR",
 "rationale": "one sentence, naming the rule you applied"}

Precedence for "status", applied in this order: CONTEXT_INCOMPLETE beats everything; then BILLED
if the carried state already says one, or if the register bills it this run; then ON_HOLD if a
formal hold is open; then REVIEW if the day count is 21 or more; otherwise UNBILLED.

"route_to": NONE for BILLED, ANALYST for CONTEXT_INCOMPLETE and UNBILLED, SUPERVISOR for ON_HOLD
and REVIEW (Rule B-8). This pack never creates, releases or posts an invoice -- "route_to" is a
recommendation for a person, not an action this reading takes (Rule B-9).

Carried state
----------------------------------------------------------------
No earlier reconciliation run has been recorded for this work order. This is its first appearance on the watch, so no invoice line has been matched yet and no earlier day count is available.

Unbilled-work reconciliation snapshot
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated unbilled-work reconciliation
snapshot for an AI use-case kit; it reproduces no real hauler, customer, contract or
billing register. The billing rule and its threshold are ILLUSTRATIVE.

Work Order
----------------------------------------------------------------
Customer                       : CUST-8021 (Grandview Retail Plaza)
Work order reference           : WO-0001
Service type                   : COMPACTOR_HAUL
Container / equipment          : CMP-6207
Route                          : RT-04
Completion date                : 2026-02-21
Contract value for this job    : 1,766.72 USD
Evidence required for billing  : driver completion confirmation AND a weight ticket

Billing Rule
----------------------------------------------------------------
Supervisor review threshold    : 21 days unbilled (operator-tunable; see .env
                                SUPERVISOR_REVIEW_DAYS)

Rule B-1  A completed work order becomes a BILLING CANDIDATE the day after its completion date,
          and stays one until an invoice line in the register genuinely references it.
Rule B-2  Days unbilled are counted in whole RECONCILIATION WINDOWS. The first window runs from the
          completion date to the first reconciliation run; every window after that is exactly the
          7-day cadence. A window in which a formal billing hold (Rule B-3) was open for ANY part
          does not count toward the total AT ALL; a window with no formal hold open at any point
          counts in full. This is a stated simplification -- a hold that closes partway through a
          window still excludes the whole window -- and it is not split into partial days.
Rule B-3  A billing hold suspends a window's count only where the Evidence Log records a DATED,
          FORMAL hold entry — opened and, once resolved, closed — or where the carried state says a
          formal hold was already open coming into this window. A standing note that SOUNDS like a
          hold (a customer disputing a weight, an account contact asking to delay invoicing) does
          NOT suspend anything unless a formal hold was actually opened for it.
Rule B-4  A work order is BILLED when the Invoice Register carries a line that genuinely references
          it: by its exact work order number, by a BUNDLED line listing several work order numbers
          for one route-week, or by a REFORMATTED job reference the billing system appends its own
          suffix to. A line that merely QUOTES this work order's number while describing a
          correction, a reversal or a charge taken back out does NOT bill it — quoting the number is
          not billing the job, and Rule B-6 is what such a line is for.
Rule B-5  Evidence completeness depends on the service type. A WEIGHT-BASED service (compactor haul,
          roll-off pull, disposal trip) needs a driver completion confirmation AND a weight ticket on
          file. A FLAT-RATE service (container swap, roll-off delivery, extra pickup) needs only the
          driver completion confirmation. Where the evidence this rule requires is not on file, the
          work order is CONTEXT_INCOMPLETE: no status band is reported and it is routed to the
          analyst to chase the evidence, never billed on a guess.
Rule B-6  A correction or reversal line in the register that names this work order while stating no
          service was performed, or that the charge was miskeyed, does not change this work order's
          status in either direction. It is evidence about the LEDGER, not about the JOB.
Rule B-7  Once a work order is reported BILLED, it stays BILLED on every later reconciliation run —
          this pack does not re-open, re-flag or re-age a billed work order, even where a later
          register shows that invoice voided or credited. Voiding is out of scope for this pack.
Rule B-8  Routing: a CONTEXT_INCOMPLETE work order routes to the ANALYST to gather evidence. A BILLED
          work order routes to NONE — nothing to do. An ON_HOLD work order routes to the SUPERVISOR,
          because a hold means a person is already deciding something. An UNBILLED work order younger
          than SUPERVISOR_REVIEW_DAYS routes to the ANALYST; at or past that age it is a REVIEW and
          routes to the SUPERVISOR.
Rule B-9  This pack never creates, releases or posts an invoice, and never marks a work order billed
          on its own authority. "BILLED" is read off the register, never asserted; every reading's
          output is a routing recommendation for a person to act on.

Billing Position
----------------------------------------------------------------
Reconciliation run             : 2026-03-02   (scheduled run 1 of this work order)
Watch cadence                  : every 7 days (Mondays, 08:00 local)
Previous reconciliation run    : -- this is the first scheduled run of this work order
Completion date                : 2026-02-21
Days since completion, as of this run : 9
Evidence on file               : driver completion confirmation ON FILE; weight ticket ON FILE

Invoice Register
----------------------------------------------------------------
Invoice INV-713778, line 2: WO-0001, compactor haul, weight-based, $1,766.72

Evidence Log
----------------------------------------------------------------
Entries raised in this window only.

2026-02-25  job re-assigned to a different analyst in the queue

Billing Desk Notes
----------------------------------------------------------------
Customer's site manager asked for an itemized breakdown before the invoice goes out.

Raw responseThe raw response

{"status": "BILLED", "days_unbilled": 9, "evidence_complete": "YES", "route_to": "NONE", "rationale": "Invoice register line INV-713778 line 2 genuinely references WO-0001 for the compactor haul, so Rule B-4 marks it BILLED with 9 days unbilled and no formal hold applies."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch waste hauls that are still unbilled — 150 reconciliation snapshots. One model answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

Every cell is compared for exact equality against a gold set computed by tools/build_corpus.py from the billing model -- the resolved status, the hold-adjusted day count, the evidence flag and the routing call -- and re-derived from src/billing.step by evals/check_labels.py before any run may spend. A null days_unbilled MATCHES a null one, because 'no count can be computed' is the correct answer on the 6 CONTEXT_INCOMPLETE readings rather than a missing value. No model grades anything, here or anywhere in this kit.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on this page is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-24. The provider that actually ran all 480 calls made while building this kit is kept out of these projection tables per this estate's naming rule for rendered pages, so nothing here is what was paid. The real spend for this kit's whole build is on the shared call ledger, not on this page.

Grading unitWhat the grading figure prices

The GRADER costs nothing: evals/scoring.py makes no model call, and neither do the three free floors it is compared against or the pre-flight. The cost of the RUN is a different figure and lives in lens 07.

The gradersOne way to grade, and why it is the only one

⚑ THE MODEL WINS EVERY MEASURED CELL, AND THE FLOOR IS NOT A STRAW MAN. b002 (broad-match-mem) reaches 96.0 pct status accuracy for $0.00 by widening its matcher to bare numbers (catching the bundled and reformatted references b000/b001 miss) and its hold check to any hold-sounding word anywhere in the notes. Both widenings that make it strong are also what fool it: 2.84 pct false ON_HOLD (the hold-decoy notes it cannot tell from a formal entry) and it is fooled by both reversal-decoy lines the same way the two weaker floors are. ⚑ THREE FLOORS, BECAUSE ONE CANNOT SEPARATE THE MATCH FROM THE MEMORY FROM THE HOLD SCOPE. b000 is the incumbent -- an exact work-order-number join with no memory and no hold awareness: 68.67 pct, missing every bundled and reformatted match and every ON_HOLD reading. b001 adds carried memory and a hold check scoped correctly to the Evidence Log's own formal entries: 91.33 pct, +22.66 points, and already reaches 100 pct hold recall with ZERO false holds -- proving the hold-decoy trap is beaten by SCOPE, not by width. b002 then widens the matcher to catch bundled/reformatted (+33 points of billed recall, 68.57 -> 100.0) but widens the hold check too far and starts catching decoys. ⚠︎ READ THE PER-BUCKET CUT, NOT THE AGGREGATE: b002's 96.0 pct aggregate hides that it is wrong on 4 of 12 hold-decoy readings and both reversal-decoy readings -- exactly the two buckets this kit was built to test, and exactly where the model's 100 pct earns its keep.

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-24. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same reading (one work order, at one reconciliation run), the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

480 live calls were attempted building this kit; all 480 returned something. 309 are the committed measurement set (9 final calibration, 150 scored, 150 stateless control) at $1.3692 projected. 171 were DISCARDED, not because a call failed but because an exploratory r001 attempt (150 calls) surfaced a genuine prompt-clarity defect -- the carried-state sentence for a BILLED work order did not restate its frozen day count, so the model correctly answered null rather than guess -- which was fixed in src/state.py and the run re-fired clean; three earlier calibration probes (21 calls total) were superseded the same way while finding the right token ceiling. Nothing here was re-fired to chase a better SCORE; the defect that was fixed is documented in Business.not_good_enough and Architecture.components, and the discarded runs' own numbers are not quoted anywhere as findings. Grading itself costs $0.00 on every arm -- evals/scoring.py is pure code.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, priced at 6x input on the shared projection card. 68.5 pct of this run's projected bill is output, even though output tokens are only 27 pct of the total token count -- what you are paying for is the model reasoning about which of five register-line shapes applies, not reading the page.
• THE CADENCE, which multiplies everything else. One reading is one call; the watch wakes weekly; the bill is rows x runs, not rows.
• THE OPEN QUEUE, not the desk's total job volume. A work order that gets billed leaves the effective watch the run after it is matched (Rule B-7 freezes it, but this kit still re-reads and re-reports it every later run rather than dropping it from the corpus -- a real deployment would retire a billed job from the watch list entirely, which this kit's own corpus does not model).

Your volumeWhat it costs at your volume

LINEAR IN WORK ORDERS x RECONCILIATION RUNS, AND THAT IS THE WHOLE WARNING. Ten times the open work orders is ten times the calls at the same cadence -- there is no batching, no cache and no early exit, because every open candidate is re-read whole on every run by design. A billing desk holding 500 open work orders on this weekly watch is 500 calls a week, about $0.39 a week on the shared projection card. Nothing about the per-call price changes; the multiplier is the schedule and the queue size.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

per-query average, this run

The codeEvery file, and what it is for

12 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pythe corpus generator — a swap seam

Generates 50 completed work orders x 3 weekly reconciliation runs = 150 snapshots from a fixed seed (SEED = 20260824), across ten customers and six service types. Plants ten explicit buckets: three ways an invoice line genuinely bills a job (exact, bundled, reformatted), one way a line only quotes a job's number without billing it (the reversal decoy), two hold shapes (closes inside the next window, spans a whole window silently), a hold-sounding note with no formal hold behind it, an evidence gap that resolves, and two plain aging trajectories. The gold labels are src/billing.step's output over the raw per-run facts, never typed by hand.

You change it to: The customers, service types, invoice-line phrasings and the seed. Keep the eight section headings or src/segment.py's assertion refuses to start.

# Generate the corpus and the answer key. Deterministic: one seed, one corpus, forever.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
SEED = 20260824
COUNT = 50
RULE = "-" * 64
RUN_DATES = (dt.date(2026, 3, 2), dt.date(2026, 3, 9), dt.date(2026, 3, 16))
CUSTOMERS = [

src/billing.pythe billing rule — a swap seam

The rule as pure code: the whole-window hold exclusion (Rule B-2/B-3), the invoice-match test (Rule B-4/B-6), the evidence-completeness test by service type (Rule B-5), and the routing table (Rule B-8). No model, no judgement. The nine rules, the 21-day review threshold and the whole-window convention are INVENTED and the kit ships that as a stated limitation -- see Data.why_this_corpus.

You change it to: The whole-window exclusion convention (Rule B-2) and what counts as a FORMAL hold (Rule B-3). A real billing desk's own hold workflow -- who can open one, what evidence it requires -- should replace the invented rule.

# The unbilled-work rule as arithmetic. Pure code, no model, standard library only.

BILLED = "BILLED"
UNBILLED = "UNBILLED"
ON_HOLD = "ON_HOLD"
REVIEW = "REVIEW"
CONTEXT_INCOMPLETE = "CONTEXT_INCOMPLETE"
STATUSES = (BILLED, UNBILLED, ON_HOLD, REVIEW, CONTEXT_INCOMPLETE)
YES = "YES"
NO = "NO"
EVIDENCE = (YES, NO)

src/state.pythe carried state — a swap seam

SEAM 2 -- the thing that makes this a monitor. Four scalars per work order (billed, hold_open, the frozen or running day count, the status last reported), written from the arithmetic and rendered into one English sentence for the prompt. ⚠︎ A real defect was found and fixed here during this kit's own measurement: the first version described a BILLED work order without restating its frozen day count, and the model correctly answered null rather than guess it -- 21 of 21 memory-dependent BILLED cells wrong. Naming the number in the sentence took memory-dependent day-count accuracy from 14.29 pct to 100 pct; see the module's own docstring for the full account.

You change it to: load()/save() are a JSON file today. Point them at a table or a billing data mart and nothing else in the kit changes -- for_workorder() and advance-by-step() are the whole interface, and describe() is the only thing the prompt sees.

# The carried state -- the thing that makes this a monitor and not another one-shot classifier.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEFAULT_PATH = os.path.join(HERE, "data", "state.json")
def load(path=DEFAULT_PATH):
def save(store, path=DEFAULT_PATH):
def for_workorder(store, workorder_id):
def describe(state):

src/segment.pythe splitter

Splits a reconciliation snapshot into its eight named sections. Pure code; evals/check_labels.py asserts all eight are present in all 150 documents before a run may spend.

# Split a reconciliation snapshot into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "Work Order", "Billing Rule", "Billing Position",
RULE = "-" * 64
def split(text):
def render(secs):

src/select.pythe selector

Decides which sections reach the model. Site Contact -- the customer site's name, direct dial and work email -- is mapped by no field and is subtracted unconditionally by _fallback(), so the one section whose subject is a PERSON never leaves the machine even when a source-system rename makes every hint match nothing.

# Pick which sections of a reconciliation snapshot are sent. Pure code -- the last deterministic

BANNER = "Synthetic Record"
WORKORDER = "Work Order"
RULE = "Billing Rule"
POSITION = "Billing Position"
REGISTER = "Invoice Register"
EVIDENCE = "Evidence Log"
CONTACT = "Site Contact"
NOTES = "Billing Desk Notes"
NEVER_SENT = (CONTACT,)

src/prompt.pythe prompt

Three parts: the fixed instruction, the carried-state sentence, and the selected sections in document order. The stateless build replaces the carried-state sentence with a line saying no history is available -- byte-identical everywhere else, which is what makes the control a control, and evals/check_labels.py asserts exactly that.

# Assemble the one prompt this kit sends. No template engine, no framework -- string

INSTRUCTION = """\
def build(text, carried, stateless=False):
SYSTEM = ("You are a scheduled unbilled-work reconciliation watch. You apply a written billing "
def build_parts(text, carried, stateless=False):

src/adapters/__init__.pythe adapter — a swap seam

SEAM 1 -- the model. Raw HTTP for every provider, standard library only. Returns input/output tokens, reasoning tokens and finish_reason beside the text, and separates a transport failure from an HTTP status so a dropped connection is retried rather than recorded as a lost reading.

You change it to: PROVIDER, BASE_URL, API_KEY and MODEL in .env. Adding a provider is one function and one entry in PROVIDERS; it must return token counts, because lens 05 publishes them and lens 07 prices them.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/monitor.pythe watch

One work order, one reconciliation run, one call. Parses identifying facts off the page with a regex (the model is never asked to compute a calendar difference), assembles the prompt, calls the model, parses the reply. Holds MAX_TOKENS = 24000, the published ceiling. Never creates, releases or posts an invoice -- see the module's own guardrail note.

# One work order, one reconciliation run, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = P.SYSTEM
MAX_TOKENS = 24000
FIELDS = ("status", "days_unbilled", "evidence_complete", "route_to")
def documents():
def chains():
def load_doc(doc_id):
def position_of(text):

src/app.pythe local UI

One work order, one reconciliation run, its carried state and its verdict, on 127.0.0.1:8208. Renders with no key. Shows the carried sentence verbatim, the free floor's answer beside the model's, and a second button that replays what run r001-unbilled-work actually answered, straight off the committed result file, labelled as a replay.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
PORT = int(os.environ.get("PORT", "8208"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-unbilled-work")
GOLD_ROWS = {r["doc_id"]: r for r in
def carried_for(doc_id):
class H(BaseHTTPRequestHandler):
def main():

evals/baseline.pythe free floors

exact-join: an exact work-order-number string join, no memory, no hold awareness -- the incumbent. exact-join-mem: the same join plus carried memory and a hold check scoped to the Evidence Log's formal entries only. broad-match-mem: carried memory, a bare-number search that catches bundled and reformatted references, and a hold check widened to any hold-sounding word in the notes. 0 calls, $0.00, all three scored through the identical scorer.

# THE FREE FLOORS. Three of them, none a strawman, all of them $0.00.

MODES = ("exact-join", "exact-join-mem", "broad-match-mem")
REVIEW_DAYS = B.SUPERVISOR_REVIEW_DAYS
HOLD_WORDS = ("hold the invoice", "disputes the", "asked us to hold", "budget resets",
SECTION_NAMES = ("Synthetic Record", "Work Order", "Billing Rule", "Billing Position",
def _num(pat, text, cast=int):
def _section(text, name):
def review(text, carried=None, mode="broad-match-mem"):

evals/scoring.pythe scorer

Exact match per cell against the computed gold, split ways an average would hide: the four fields, BILLED recall against a false-BILLED rate, ON_HOLD recall against a false-ON_HOLD rate, under- versus over-escalation counted apart, the memory-dependent subset, and every bucket that plants a trap, cut separately. No judge model.

# Score a run against the answer key. Pure code, exact match per cell. No model grades anything.

FIELDS = ("status", "days_unbilled", "evidence_complete", "route_to")
BUCKET_CUTS = ("billed_exact", "billed_bundled", "billed_reformatted", "unbilled_watch",
def _pct(n, d):
def _days(v):
def score(records, golds):

evals/check_labels.pythe pre-flight

Eighteen things that must be true before a run may spend: all eight sections parse, every work order's run sequence is complete and gap-free, no run-varying section restates an earlier match, neither a formal-hold flag nor a final day count is printed anywhere on the page, an exact join is proven to miss every bundled and reformatted reference AND be fooled by both reversal lines, the hold-decoy bucket is proven to carry no formal hold entry, the privacy guard holds and would fail without it, the two prompt builders differ on exactly one line, no code path creates, releases or posts an invoice, and the answer key replays from src/billing.step.

# Everything that must be true BEFORE a run is allowed to spend. Free, and it runs in seconds.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
FAILED = []
def check(name, ok, detail=""):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2454 input and 891 output tokens per reading (one work order, at one reconciliation run), all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

No injection or red-team run was fired against this kit. Billing Desk Notes is this kit's one injection surface -- free text a billing analyst types, sent to the model verbatim -- and one of the shipped notes is instruction-shaped by design ('Customer's accounts-payable contact asked us to hold off invoicing until next quarter's budget resets.'), matching the convention sibling kits in this series use, but its effect on a live model's status or route_to call was never measured.

API_KEY is read from .env (gitignored) or the real environment; never logged, never placed in a prompt, and redacted from any provider error message src/app.py passes to the browser (exact-string replace on api_key and base_url before the message is returned).

The experimentWe did NOT attack it -- and this is the surface we left open

An indirect prompt injection needs a field somebody outside the process can write into, and this kit's is the Billing Desk Notes -- free text a billing analyst types, sent to the model verbatim. One of the shipped notes is instruction-shaped by design, but no adversarial trial was fired against it. Both were run for real on 2026-08-24.

No boundary here is measured in both directions. The one gate below reports the banned-code-path scan, which is a static check on what the code CAN do, not a measurement of what the model did under attack.

The resultNot measured. No adversarial trial was run against the Billing Desk Notes injection surface, including the instruction-shaped note already shipped in the corpus.

No red-team run was attempted for this kit. evals/check_labels.py's banned-code-path scan (0 violations) -- the only automated gate this kit runs before spending. Cited for reference only, same corpus and model as r001-unbilled-work.

HonestyWhat this does not prove

• Whether the instruction-shaped note ('asked us to hold off invoicing until next quarter's budget resets') actually suppresses or changes route_to on a live model -- named as a surface, never probed.
• Whether the model would follow an injected instruction to falsely report BILLED, or a different status, on a job that is neither -- never tested.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never create, release, post or void an invoice under any authority, and never present the 21-day threshold or hold convention as any real hauler's actual policy.

Stated to the model on every call in src/prompt.py's INSTRUCTION block (Rule B-9), and enforced mechanically by evals/check_labels.py's banned-code-path scan, which greps every .py/.js file in the kit for the names of a release/post/create-invoice path before any run may spend.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is a PROMPT RULE PLUS A STATIC-ANALYSIS SCAN, not a runtime enforcement layer. Nothing in the code stops a forker from adding a release_invoice() function tomorrow; the scan only catches it the next time someone runs evals/check_labels.py, a manual step, not a hook that fires on every commit.

WatchedWhat is watched, and why that one

The kit measures 39 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

20 measured by the latest run 19 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (analyst_enough_cells 120, billed_cells 35, cadence_days 7, context_incomplete_cells 6, documents 150, exposure_usd_gold 83399.01, hold_cells 9, memory_cells 21, needs_supervisor_cells 30, not_billed_cells 115, not_hold_cells 141, readings_scored 150, stateless False, supervisor_review_days 21, workorders 50) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

7 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

triage · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 3 runs measured the same system — they differ on floor, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

triage · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on analyst_enough_cells, billed_cells, context_incomplete_cells, documents, exposure_usd_gold, hold_cells, memory_cells, needs_supervisor_cells, not_billed_cells, not_hold_cells, readings_scored, stateless, workorders — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 20 chips that all say so.

DeviationsWhat deviated

0 breaches across 7 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Checked on every run via evals/check_labels.py's banned-path scan, run manually immediately before any paid call -- not on a schedule and not on a git hook.

In one lineNo framework, deliberately

NO FRAMEWORK. The same position as every kit in this series: a folder of readable Python, no framework dependency, and a prompt anyone can read end to end in src/prompt.py. LangChain/LlamaIndex-style abstractions would own a retrieval step -- there is none here, the whole snapshot goes into one prompt -- and the memory/checkpoint layer, which is already the entire surface of src/state.py's four scalars.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear: src/monitor.py -> src/prompt.py -> src/adapters -> evals/scoring.py. No branching, no tool-calling and no agent loop -- a framework's graph/DAG abstraction has nothing to route here.

The other sideWhat a framework costs you

• The cost of not using a framework is that swapping providers means editing src/adapters/__init__.py's PROVIDERS dict by hand (one function, one dict entry) rather than swapping a framework's provider string. In exchange the prompt has no hidden templating layer and requirements.txt stays empty.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-unbilled-work on the same tier, memory removed (THE CONTROL), 2026-08-24. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 5 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

3 runs not plotted. b000-unbilled-work-exactjoin, b001-unbilled-work-exactjoinmem, b002-unbilled-work-broadmatch recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the population is re-read whole on each scheduled run, and the prior run's carried state is what makes a reading a CHANGE.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved; every customer, work order, container, invoice line, evidence-log entry and desk note is invented here. Verified against the repository's own LICENSE file on 2026-08-24. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe status, day count, evidence flag and routing call, per reading, exact match against the computed answer key

For each of the 150 readings and each of the four answered fields, did the reply equal the computed answer key? Status, evidence-complete and route_to are compared exactly against closed lists; days_unbilled is compared as a whole number, and a null MATCHES a null because 'no count can be computed' is the correct answer on 6 readings and must not be scored as a parse failure.

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model. The same function evals/baseline.py is scored through.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 150 per field. A reading whose reply did not parse counts as a MISS in every field, never as an exclusion.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, computed by src/billing.step over the planted inputs at generation time. This grader IS the reference, so its own TPR/TNR are not measured -- scoring it against itself would be circular.

How tight can the band be? No tuned threshold anywhere on the model path -- the reply decides. Denominators are printed beside every rate because three are thin: 35 BILLED cells, 9 ON_HOLD cells, 6 CONTEXT_INCOMPLETE cells, 2 reversal-decoy readings.

Cadence: Re-run evals/check_labels.py (free) on any change to the corpus generator, src/billing.py, src/segment.py or src/select.py. Re-run the scored eval (paid) on any change to src/prompt.py or src/state.py.

The decisionWhen to reach for it