The business caseThe problem this solves

'Which scale tickets are heavy this week' is a SELECT and nobody needs a language model for it. The question a disposal recon desk actually has to answer is the next one: is this reading a CHANGE, and is it a change somebody has already raised? A route tipping 5.51 tons against a 4.34-ton norm is a routine watch item on its first week, an escalation on its second, and on its third it is a route whose norm has moved -- and if a dispute is already open with the facility, raising it again is a second copy the facility rejects and somebody reworks by hand. None of those three facts is on the sheet, because the sheet records what the FACILITY did in the week it covers and the monitor's own raise happened after that week closed. Somebody working a weekly disposal recon queue by hand: opening every open route, adding up the week's scale tickets, comparing them against the norm they believe is in force, then going back through the last month's exception queue to work out whether this route has been drifting for weeks and whether a dispute is already sitting with the facility. The first three steps are a spreadsheet; the fourth is the one that is skipped when the queue is long, and skipping it is what files the same dispute twice.

The inputThe actual weekly monitor sheets

The corpus is 150 weekly monitor sheets, 0.60 MB (json 1 · jsonl 1 · txt 150). A DISPOSAL WATCHLIST IS A LIST OF CUSTOMER PREMISES. Not routes in the abstract -- addresses: which business, at which site, producing how much waste, on which days. That is competitive information about the CUSTOMER, not about the hauler, and an internal recon file is exactly the sort of working document that gets pasted into a chat window on the way to a supervisor. So the corpus carries a full Customer Contact section on every sheet -- holder, service address, telephone, email -- specifically so that the guard which withholds it has something real to withhold and can be red-proven. IT IS SYNTHETIC BECAUSE THE MEASUREMENT REQUIRES A HISTORY NOBODY WOULD GIVE US. The whole question is what six carried scalars are worth, and answering it needs routes whose correct answer provably CANNOT be derived from the week in front of you: a streak already running, an anomaly already open, a norm already rebased. A real book would have all three and no way to prove which cells they govern; here 64 of 150 cells are marked memory-dependent by running every week TWICE, once from its real carried state and once from a blank one, and taking the disagreements. THE DISTRIBUTION IS DELIBERATE AND STATED. 73 of 150 weeks are quiet (CLEAR or NOT_MONITORED), 77 are on the watchlist, and 32 have ALREADY_OPEN as their correct answer -- the single cell no one-sheet reader can reach. A corpus that was 90 pct quiet would let an always-CLEAR arm score 90 pct, which is why the always-CLEAR arm is published as its own floor at 46.67 pct rather than left as an argument.

Synthetic Record
----------------------------------------------------------------
This is a generated disposal weight monitor sheet. Every route, ticket, tonnage,
customer and address in it is invented. It reproduces no real haul, no real facility
and no real account, and the rule reproduced below is illustrative only.

Route
----------------------------------------------------------------
Route                     : RTE-0001
Service week              : 2026-03-07 to 2026-03-13
Facility                  : Harrow Creek MRF
Container                 : 20 yd roll-off
Service days              : Tue, Fri
Segment                   : retail plaza
Book norm as first set    : 8.49 tons per service week
Book band                 : 7.22 to 9.76 tons  (plus or minus 15 per cent)
On the disposal book      : yes

Weight Rule
----------------------------------------------------------------
  1. Each route carries a NORM, in tons of net disposal weight per service week, and a tolerance
     band of plus or minus 15 per cent of that norm. The week's total net tonnage from the scale
     tickets is compared against the band of the norm IN FORCE, which is the norm the monitor is
     currently using and is not always the book norm printed on this sheet.
  2. A week whose total falls outside the band is an OUT week, and its DIRECTION is HIGH or LOW.
     WEEKS OUT is the number of consecutive out weeks in the SAME direction ending with this one.
     A week inside the band has weeks out 0. A direction that changes restarts the count at 1.
  3. The count is RESET by a disposal exception this week's Open Items log records as CLOSED --
     the ticket was corrected, a credit was issued, or the route was re-normed. A week carrying a

The outcomeWhat a good result looks like

Per route per week: the verdict after the already-open rule has been applied, how many consecutive weeks the route has been outside its band, whether the norm itself should be rebased, and which class the recorded reason maps to -- plus a one-sentence rationale naming the rule applied. A watchlist row, not an action.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• A book where nothing has ever been disputed, every route's norm is current, and no route has been out of band two weeks running — the free desk band, at $0.00
  With no streaks and no open items there is nothing to carry. The floor gets the verdict right on every in-band week and every first excursion for nothing, and it TIES the model on anomaly class (90.67 pct each) across the whole corpus.
• A weekly disposal recon queue that carries open disputes and re-normed routes -- the ordinary state of a real book after a quarter — this kit, with the carried state
  100.00 pct verdict against the floor's 63.33 pct, 100.00 pct against 31.17 on the 77 watchlist weeks, and ZERO duplicate disputes against the floor's 27.
• You have no history at all -- a new book, or a monitor being stood up — the free desk band for the first two weeks, then this kit
  Rule 7 says a route with fewer than 2 recorded weeks has no observed norm, and the corpus's 15 new_route weeks are answered correctly by both arms. Memory is worth nothing until there is some.
• Sizing what this costs on a real disposal book — read Cost.cost_at_10x before anything else
  At $0.005219 a route-week on the projected card, a book of 20,000 routes checked weekly is about $5427 a year, and the free floor already gets 63.33 pct of the verdicts right. The honest deployment is the subset the floor cannot answer -- routes with an open item or a live streak -- not the whole book.

And where nothing here is good enough:

• You need a confident cause code on every flagged week for a downstream workflow — neither, without deciding what a hedge means first
  The two arms tie at 90.67 pct and fail on disjoint cells. The floor gives you a confident topical code where nobody recorded a cause; the model gives you CAUSE_UNKNOWN where a dispatcher hedged. Which is worse is your call and this page will not make it.
• Deciding whether to rebase a route's billed norm — neither, without a human step
  NORM_SHIFTED is the highest-consequence output this kit has -- it recommends changing what a customer is billed against -- and it is the one field the model gets wrong, always in the direction of recommending a rebase that is no longer warranted.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-23. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-23 — r001-ticket-anomaly. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with NO key configured reproduces the corpus byte-identically (python3 -m tools.build_corpus), passes all eight pre-flight assertions (python3 -m evals.check_labels), red-proves them (--self-test, 4 of 4 convictions), runs both free floors and the wiring stub end to end, and serves the UI on port 9005 with the committed r001 answers replayed beside the free floor. Nothing above needs a credential, a network call or an install: requirements.txt names nothing, and the only imports anywhere in the kit are json, os, re, sys, time, random, datetime, argparse, collections, concurrent.futures, urllib, http.client and http.server.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per weekly sheet: one HTTP request carrying the assembled prompt, and the reply parsed to a verdict, a week count, a norm status and a class. Measured over 150 weeks on r001-ticket-anomaly, 50 route chains wide with 12 concurrent workers; the whole pass took 194.4 seconds of wall clock. The p95 of 50568 ms is not a slow network -- it is the provider reasoning: 93.6 pct of this run's output tokens were provider-side reasoning, and the longest reply ran to 14904 tokens.

Current processWhat it replaces

Somebody working a weekly disposal recon queue by hand: opening every open route, adding up the week's scale tickets, comparing them against the norm they believe is in force, then going back through the last month's exception queue to work out whether this route has been drifting for weeks and whether a dispute is already sitting with the facility. The first three steps are a spreadsheet; the fourth is the one that is skipped when the queue is long, and skipping it is what files the same dispute twice.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• 4 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one routes (3 weekly runs each) is

No split, and no chunking. The unit is a ROUTE -- a run of three consecutive weekly monitor sheets processed strictly in order, because week 3's prompt carries a streak week 2 produced. Routes are independent and run concurrently; weeks within a route never do. evals/check_labels.py asserts all 50 sequences are complete and gap-free before a run may spend.

SetupWhat the setup figure measured

There is no index and no retrieval step. tools/build_corpus.py writes 150 documents and the answer key from a fixed seed with no clock read and no model call, in well under a second, and the monitor re-reads the whole population on each scheduled run rather than searching it.

LicenceLicence

MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved; every route, ticket, tonnage, facility, customer, address and recorded reason is invented by the generator. See kits/UC0088-ticket-anomaly/data/SOURCES.md.

Bring your ownBring your own weekly monitor sheets

Replace src/norms.py FIRST, and not as an option. The band, the raise threshold, the norm-shift threshold and the dispute window shipped here are invented; yours are in your facility's published terms and your own hauling contract. Then replace data/corpus/ with your sheets and data/gold.jsonl with your labels -- the seven section headings in src/segment.py are the whole contract, and nothing else in the kit reads the text. Everything else transfers unchanged, including the two floors, because both are computed from whatever rule src/norms.py holds.

⚠︎ And what stops being true when you do: Every figure here is a property of a three-week corpus of 50 invented routes, measured once. Three denominators are small: 3 NOT_MONITORED cells, 9 flip cells, 15 closed_reset cells. Three weeks is long enough to plant a streak that reaches the norm-shift threshold and NOT long enough to test what happens on week twelve of a route that has been out of band since spring -- the accrual is constant-cost by construction, but nobody here has run it that far. And the whole corpus is one facility per route, one norm per route and no seasonality model; a real book has all three.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 7,233 characters assembled, 4,434 are contexts — 61% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Replayed from the kit's own src/prompt.py::build on RTE-0034-W1 with the exact carried state r001-ticket-anomaly recorded for that call, read out of the run file's per_week block rather than re-derived. 7404 characters including the system message.

Full promptThe prompt in full

You apply a written disposal weight rule to one waste route for one service week. You answer with one JSON object and no other text.

You are running one weekly disposal weight monitor over one waste collection route, for one service
week, and deciding whether this week's scale tickets are an anomaly that should be raised with the
disposal facility.

The weight rule, the tolerance band, the raise threshold and the norm-shift threshold are reproduced
on the sheet below. Apply them exactly as written. Rules 2, 3, 4 and 7 depend on what earlier weeks
did, which you cannot see; what is known is stated under "Carried state" and is the only history
available to you. Do not assume anything about earlier weeks beyond it.

How to decide:
 - Band this week's TOTAL NET TONS against the norm IN FORCE from the carried state, not against the
   book norm printed on the sheet, wherever the two differ.
 - WEEKS OUT counts consecutive out weeks in the same direction, ending with this one, and it
   INCLUDES the weeks the carried state reports. A week inside the band is 0. A direction that
   changes restarts the count at 1.
 - A CLOSED entry in this week's Open Items log resets the count: that week counts as the first of
   any new run, and it also closes an anomaly the carried state says is open.
 - An anomaly the carried state says is STILL OPEN is not raised again. While the route stays out of
   band with that item open, the verdict is ALREADY_OPEN, whatever the week count reaches.
 - A route not on the disposal book is NOT_MONITORED, weeks out 0, anomaly class NOT_APPLICABLE.
 - A week inside the band is CLEAR and its anomaly class is NOT_APPLICABLE.

Answer with a single JSON object and nothing else:

{"verdict": "CLEAR|WATCH|RAISE|ALREADY_OPEN|NOT_MONITORED",
 "weeks_out_of_band": <whole number of consecutive out weeks ending with this one>,
 "norm_status": "NORM_HOLDS|NORM_SHIFTED|NORM_UNKNOWN",
 "anomaly_class": "TARE_DRIFT|ROUTE_MISPOST|SERVICE_STEP|CONTAMINATION|EXTRA_PULL|SEASONAL_NORMAL|CAUSE_UNKNOWN|NOT_APPLICABLE",
 "rationale": "one sentence, naming the rule you applied and the week count you reached"}

"verdict" is the verdict AFTER the already-open rule has been applied, which is not the verdict the
week count alone would imply. "norm_status" is NORM_UNKNOWN whenever the carried state reports fewer
than 2 recorded weeks on the monitor, whatever the tonnage does; it is NORM_SHIFTED once the route
has been out of band in the same direction for 3 or more consecutive weeks. "anomaly_class" is the
recorded reason mapped to one of the codes above; use CAUSE_UNKNOWN when the recorded reason maps to
none of the topical codes -- including where the note names a possible cause and then says nobody
established or recorded it -- rather than choosing the nearest.

Carried state
----------------------------------------------------------------
The norm IN FORCE for this route is 4.34 tons per service week. It was rebased in an earlier run and is NOT the book figure printed on the sheet; band this week against 4.34 tons. It has 30 recorded week(s) on the monitor. As at the previous run it had been out of band HIGH for 2 consecutive week(s). A disposal anomaly raised by this monitor is STILL OPEN against this route. It was opened in an earlier run and no sheet records it. It was reported RAISE last run.

Weekly monitor sheet
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
This is a generated disposal weight monitor sheet. Every route, ticket, tonnage,
customer and address in it is invented. It reproduces no real haul, no real facility
and no real account, and the rule reproduced below is illustrative only.

Route
----------------------------------------------------------------
Route                     : RTE-0034
Service week              : 2026-03-07 to 2026-03-13
Facility                  : Cedar Ridge Landfill
Container                 : 30 yd roll-off
Service days              : Tue, Fri
Segment                   : multi-family
Book norm as first set    : 3.34 tons per service week
Book band                 : 2.84 to 3.84 tons  (plus or minus 15 per cent)
On the disposal book      : yes

Weight Rule
----------------------------------------------------------------
  1. Each route carries a NORM, in tons of net disposal weight per service week, and a tolerance
     band of plus or minus 15 per cent of that norm. The week's total net tonnage from the scale
     tickets is compared against the band of the norm IN FORCE, which is the norm the monitor is
     currently using and is not always the book norm printed on this sheet.
  2. A week whose total falls outside the band is an OUT week, and its DIRECTION is HIGH or LOW.
     WEEKS OUT is the number of consecutive out weeks in the SAME direction ending with this one.
     A week inside the band has weeks out 0. A direction that changes restarts the count at 1.
  3. The count is RESET by a disposal exception this week's Open Items log records as CLOSED --
     the ticket was corrected, a credit was issued, or the route was re-normed. A week carrying a
     CLOSED entry counts as the first week of any new run, whatever came before it.
  4. A route with an anomaly ALREADY OPEN against it is not raised again. Report ALREADY_OPEN for
     as long as the route stays out of band with that item open. A second dispute against the same
     route is rejected by the facility and both copies are then reworked by hand.
  5. A route not on the disposal book this week -- service suspended, container pulled, account on
     hold -- is NOT_MONITORED, weeks out 0, anomaly class NOT_APPLICABLE, and is not on the
     watchlist.
  6. Otherwise the verdict is RAISE at 2 or more consecutive out weeks, WATCH at exactly 1, and
     CLEAR at 0. A week in band is CLEAR and its anomaly class is NOT_APPLICABLE.
  7. The norm has SHIFTED once the route has been out of band in the same direction for 3 or more
     consecutive weeks: the tonnage is the new normal and the norm must be rebased, not disputed.
     A route with fewer than 2 recorded weeks on the monitor has no observed norm at all and its
     norm status is NORM_UNKNOWN, whatever the week's tonnage does.
  8. The facility's window to correct a scale ticket is 30 days from the tip date. A week not
     reviewed inside it can no longer be corrected on the ticket and becomes a credit memo against
     a customer invoice that has already gone out.

Scale Tickets
----------------------------------------------------------------
  date        ticket      gross     tare      net
  2026-03-08  TK-768519     10.71     8.23     2.48
  2026-03-11  TK-771584      9.87     8.23     1.64
  2026-03-14  TK-198948      9.62     8.23     1.39
  Total net this week       : 5.51 tons
  Tickets this week         : 3
  Oldest tip date           : 2026-03-08
  Days left in the facility dispute window : 20

Open Items
----------------------------------------------------------------
  no disposal exception was opened or closed against this route in the service
  week this sheet covers.

Route Notes
----------------------------------------------------------------
  reason as recorded : "the account downsized to a smaller yard box and the smaller unit is now in place"
  dispatcher note    : flagged in the weekly disposal pack

Raw responseThe raw response

{"verdict": "ALREADY_OPEN", "weeks_out_of_band": 3, "norm_status": "NORM_SHIFTED", "anomaly_class": "SERVICE_STEP", "rationale": "Rule 4 already-open applies: this week's 5.51 tons is above the in-force norm band (3.689 to 4.991 tons) and the carried open anomaly remains open with no reset, giving 3 consecutive HIGH out weeks (2 carried + this one) and a shifted norm."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Spot waste routes dumping more weight than usual — 150 weekly monitor sheets. One model answered, and every answer was then graded Five different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

There is no LLM-as-judge in this kit and that is not a shortcut. Three of the four answered fields are closed sets -- five verdicts, three norm states, eight class codes -- and the fourth is a whole number of weeks. A judge is for answers whose correctness is a matter of reading; these are matters of equality and arithmetic, and asking a model to grade RAISE == RAISE would add cost, variance and a second thing to be wrong. The answer key itself is computed by src/norms.replay over what the generator planted, and evals/check_labels.py re-derives all 150 rows from the shipped text before a run may spend.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran all 348 billed calls is kept out of these tables per this estate's naming rule, so nothing here is what was paid. The real spend for this kit's whole build is recorded in the shared call ledger, not on this page.

Grading unitWhat the grading figure prices

Grading calls nothing. A measured $0.00, not an unpriced one -- evals/scoring.py, both free floors, the wiring stub and the eight-check pre-flight are all pure Python and make no request of any kind.

The gradersFive ways to grade

READ THIS ROW BEFORE THE STATELESS CONTROL, BECAUSE IT IS THE STRONGER NO-MEMORY ARM AND ON TWO COMPARISONS IT WINS. ⚑ IT TIES THE MODEL EXACTLY ON ANOMALY CLASS -- 90.67 pct against 90.67 pct -- AND THE 14 CELLS EACH GETS WRONG DO NOT OVERLAP AT ALL. Not one. The floor's keyword table over-commits: 10 of its 14 answer a topical code (EXTRA_PULL, TARE_DRIFT, SERVICE_STEP) on a note that names a possible cause and then says nobody established it. The model under-commits: all 14 of its misses answer CAUSE_UNKNOWN on a note that states a cause with a hedge -- 'a mis-post from the adjacent commercial route is SUSPECTED'. Same headline, opposite error direction, disjoint cells. Which one you want depends entirely on whether a wrong confident code or a shrug costs you more, and this page will not average them into a preference. ⚑ AND IT BEATS THE STATELESS MODEL ON NORM STATUS BY 65.33 POINTS -- 72.00 pct against 6.67 pct. The floor assumes the printed norm holds, and 108 of 150 weeks in this corpus are NORM_HOLDS, so it is right by construction on the majority class. A model told it has no history correctly declines to assert an observed norm and answers NORM_UNKNOWN almost everywhere. The floor is not smarter here; it is luckier, and only against this distribution. Change the corpus to 50 pct shifted norms and the floor halves while the stateless arm does not move. WHERE IT LOSES IT LOSES BADLY AND FOR ONE REASON. Verdict 63.33 pct against 100.00 pct; on the 77 watchlist weeks, 31.17 pct against 100.00 pct; on the 64 memory-dependent cells, 32.81 pct against 100.00 pct. And the cost with a name: it files 27 DUPLICATE DISPUTES -- 84.38 pct of the 32 weeks whose correct answer is ALREADY_OPEN -- because no sheet anywhere in this corpus says an item is open. It also reports 389.21 tons raised with the facility against a true 146.14, a 166.33 pct overstatement of the disputed accrual.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same route-week, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

386 lines are on the shared ledger for this kit and 348 of them reached a provider: 6 calibration at an 8,000-token ceiling (c000, which lost one reply and is committed anyway because a ceiling that failed is a measurement), 6 more at 24,000 (c001), 150 scored, 150 the stateless control, and 36 the repeat probe. A further 38 lines -- a first attempt at that same probe, plus two single-call checks -- returned 402 Insufficient Balance from the shared account and were billed NOTHING; they are counted as discarded_calls at $0.00 because the ledger records a line before each request, and the successful probe was written to its own path rather than over the failed one so the discarded run keeps its own totals. Everything else -- both free floors, the wiring stub, the scorer and the eight-check pre-flight -- is pure code and costs $0.00. Figures are projected onto the Google Gemini 3 Flash card, not the real spend.

Cost driversWhat actually moves the bill

• PROVIDER-SIDE REASONING, 93.6 pct of output tokens (202520 of 216438) and therefore most of the bill. It was left at the provider's default and this kit has not measured what turning it off does to the answers.
• HOW MANY RULES THE WEEK TRIPS, NOT HOW LONG THE SHEET IS. Every sheet is between 4008 and 4291 bytes -- within 7 pct of each other -- and output tokens ran from 268 to 14904, a 55.6x spread on a corpus of one shape. The long replies are the routes where a rebased norm, a live streak, a direction change and a CLOSED entry all have to be resolved before a verdict falls out.
• WHETHER THE PROMPT CARRIES THE STATE -- AND HERE IT IS A COST, NOT A SAVING. Three sentences add 49 input tokens a week and 1.31x the output, because a model with a streak and an open item has four rules to reconcile and a model without them has one comparison to make. It costs 1.25x to be right.
• THE FIXED HEAD OF THE PROMPT. The system message and the instruction are 2799 of 7404 characters -- 38 pct, byte-identical on all 150 calls. Add the weight-rule block reproduced on every sheet and the route's own numbers are much the smaller half of what is sent.

Your volumeWhat it costs at your volume

Linear in ROUTE-WEEKS, and flat in history length, which is the unusual half. Each week is one call whose input is one sheet plus six carried scalars, so a route on its fortieth weekly run costs exactly what one on its third costs. 150 route-weeks project to $0.7828 on the shared rate card, so ten times the set is about $7.83. ⚠︎ AND A REAL DISPOSAL BOOK IS WHERE THIS ARITHMETIC STOPS BEING COMFORTABLE. At $0.005219 a route-week, 20,000 routes checked every week is about $104 a week and $5427 a year -- every year -- and the free desk band already gets 63.33 pct of the verdicts right for nothing. The honest reading is that this belongs on the SUBSET the floor cannot answer: routes carrying an open item or a live streak. In this corpus that is 64 of 150 weeks, which would cut the bill by about 57 pct. WHAT DOES NOT SCALE IS WALL CLOCK WITHIN A ROUTE: weeks are strictly serial. 50 chains ran 12-wide in 194.4 seconds; widening the pool is the only lever, and the depth is fixed at however many periods you replay.

Model choiceWhy this model

It is the tier the shared key is pointed at, and it is the cheap one -- the right place to start a question whose honest answer might be 'free code already does most of this', which on the anomaly-class field it does, exactly as well as the model.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every stateful number on these pages: 150 route-weeks, one completion call each, one tier. The 150-call stateless control and the 12 calibration calls are accounted for separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

14 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pythe corpus generator

Generates 50 routes x 3 consecutive weekly monitor sheets = 150 documents from a fixed seed (SEED = 20260823), across nine planted patterns, and writes the answer key by running src/norms.replay over what it planted. Nothing is labelled by hand.

# Generate the 50-route, 150-week disposal weight corpus and its computed answer key.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
SEED = 20260823
RULE = "-" * 64
FACILITIES = ["Northbay Transfer Station", "Cedar Ridge Landfill", "Pike Street Transfer",
CONTAINERS = ["4 yd front-load", "6 yd front-load", "8 yd front-load", "20 yd roll-off",
DAYSETS = [("Mon", "Thu"), ("Tue", "Fri"), ("Wed",), ("Mon", "Wed", "Fri"), ("Tue", "Thu")]

src/norms.pythe weight rule and its arithmetic — a swap seam

The band, the streak, the reset, the already-open rule and the norm-shift rule, as pure code. It is the answer key's source and the runtime never calls it to decide anything the model is asked.

You change it to: The band, the raise threshold, the norm-shift threshold and the dispute window are four constants in one file. Change them and re-run tools/build_corpus.py; the answer key follows automatically because it is computed from them.

# The disposal weight rule, and the arithmetic it implies. Pure code, no model.

BAND_PCT = 15
RAISE_AT = 2
SHIFT_AT = 3
HISTORY_FOR_NORM = 2
DISPUTE_WINDOW_DAYS = 30
CLEAR = "CLEAR"
WATCH = "WATCH"
RAISE = "RAISE"
ALREADY_OPEN = "ALREADY_OPEN"

src/state.pythe carried state — a swap seam

Six scalars between runs -- the norm in force, the consecutive-week count, the direction, whether an anomaly is open, how many weeks the route has been monitored, and the previous verdict -- plus the English they become in the prompt. Written from the arithmetic, never from a model reply.

You change it to: Six scalars and one describe() that turns them into English. Adding a seventh is one field and one clause -- and costs a handful of input tokens a week, not a growing transcript.

# The carried state -- the thing that makes this a monitor and not another one-shot classifier.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEFAULT_PATH = os.path.join(HERE, "data", "state.json")
def load(path=DEFAULT_PATH):
def save(store, path=DEFAULT_PATH):
def for_route(store, route_id, book_norm):
def describe(state, book_norm=None):

src/segment.pythe section splitter

Splits a sheet into its seven named sections. Asserted over all 150 documents by evals/check_labels.py before a run may spend.

# Split a weekly monitor sheet into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "Route", "Weight Rule", "Scale Tickets", "Open Items",
RULE = "-" * 64
def split(text):
def render(secs):

src/select.pythe section filter — a swap seam

Decides what is sent. Customer Contact is mapped by no field and is subtracted unconditionally, so it never reaches a provider on any document, for any field, including through the fallback.

You change it to: SECTION_HINTS plus NEVER_SENT. Withholding another section is one tuple entry and evals/check_labels.py re-proves it over all 150 documents.

# Pick which sections of a weekly monitor sheet are sent. Pure code -- the last deterministic step

BANNER = "Synthetic Record"
ROUTE = "Route"
RULEBLOCK = "Weight Rule"
TICKETS = "Scale Tickets"
OPENITEMS = "Open Items"
NOTES = "Route Notes"
CONTACT = "Customer Contact"
NEVER_SENT = (CONTACT,)
SECTION_HINTS = {

src/prompt.pythe prompt

Three parts: the question and the JSON shape, the carried-state block, the sheet. The stateless build replaces the middle block and changes nothing else -- asserted, not asserted-by-comment.

# Assemble the one prompt this kit sends. No template engine, no framework -- string concatenation

INSTRUCTION = """\
def build(text, carried, book_norm=None, stateless=False):

src/monitor.pyone weekly run

One sheet, one call, one parse. Holds MAX_TOKENS = 24000, the published ceiling.

# One weekly run over one route, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = ("You apply a written disposal weight rule to one waste route for one service week. "
MAX_TOKENS = 24000
FIELDS = ("verdict", "weeks_out_of_band", "norm_status", "anomaly_class")
def documents():
def routes():
def load_doc(doc_id):
def position_of(text):

src/adapters/__init__.pythe provider adapter — a swap seam

Raw HTTP over stdlib for any OpenAI-compatible provider or Anthropic. Retries transient statuses and transport failures; never retries a 4xx that will fail identically.

You change it to: .env plus the same run again. Any OpenAI-compatible provider or Anthropic.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/budget.pythe spend guard

Counts calls, not dollars, against a cap shared by every kit under the repo root. Checked once per completion, before the request.

# A cap on live calls, shared by every kit on this machine. No dependency, no service.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
KIT = os.path.basename(HERE)
ROOT = os.path.dirname(os.path.dirname(HERE))
SHARED_ENV = os.path.join(ROOT, ".env")
LEDGER = os.path.join(ROOT if os.path.exists(SHARED_ENV) else HERE, ".calls-ledger.jsonl")
PAID_ARMS = ("r", "x")
FREE_ARMS = ("t", "b")
BOOT = os.urandom(6).hex()
PID = os.getpid()

evals/baseline.pythe free floors

Two of them: the +/-15 pct desk band computed from ONE sheet, and the majority-class always-CLEAR arm. Both are pure code and cost $0.00.

# The free, no-model floors: the whole answer computed from ONE sheet, with no memory.

DESK_RAISE_DEVIATION_PCT = 30
def read_sheet(text):
def deskband(text):
def allclear(_text):
MODES = {"deskband": deskband, "allclear": allclear}
def review(text, mode="deskband"):

evals/scoring.pythe scorer

Six graders, pure code, no judge. Verdict / week count / norm status / class per week; watchlist and false alarms counted apart; duplicate raises counted on their own; the memory subset; the tonnage reconciliation.

# Score a run against the computed gold. Pure code, no model, no judge.

FIELDS = ("verdict", "weeks_out_of_band", "norm_status", "anomaly_class")
QUIET = tuple(v.upper() for v in N.QUIET_VERDICTS)
def _pct(n, d):
def _norm(v):
def _as_int(v):
def score(records, golds):
def _by_pattern(vrows):
def compare(stateful, stateless):

evals/check_labels.pythe pre-flight

Eight checks over the shipped text, run before any run may spend, with a --self-test that corrupts the corpus four ways and requires convictions.

# Everything this kit ASSERTS about its own corpus, checked before a run is allowed to spend.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
def gold_rows():
def sections_present(docs, rows):
WITHHELD = ("Customer Contact",)
def contact_never_sent(docs, rows):
def streak_not_on_the_sheet(docs, rows):
def gold_reproduces(docs, rows):
def sequences_complete(docs, rows):

evals/run.pythe harness

Routes run concurrently; the three weeks inside a route are strictly serial, because week 3's prompt contains a streak week 2 produced.

# Run the weekly monitor over the 50 routes and write one result file. THIS SPENDS MONEY.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
def load_gold():
def observation(g):
def opening_state(route_rows):
def stub_complete(cfg, system, user, max_tokens=1024):
def main():

src/app.pythe local UI

http.server on port 9005. Renders with no key and replays the committed r001 answer for the selected sheet, labelled as a replay, beside the free floor.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
PORT = int(os.environ.get("PORT", "9005"))
def _gold():
GOLD_ROWS = _gold()
RECORDED_RUN = "r001-ticket-anomaly"
def _recorded():
RECORDED = _recorded()

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 1779 input and 1442 output tokens per route-week, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

No untrusted external input surface in the shipped version. Every field that reaches the prompt comes from tools/build_corpus.py and a fixed pool of 24 generated sentences. The Route Notes are the field an outside party WOULD influence in a real deployment -- a dispatcher types them, and on a commercial book a customer can dictate what goes in them -- and this kit sends them deliberately rather than hiding its own injection surface. The one section that is never sent is Customer Contact, and that is a subtraction in code rather than a hint that happens not to match.

The provider key lives only in .env (gitignored) or the shared repo-root .env, read by src/config.py and never written to a result file, a log line, a screenshot or a page. src/app.py redacts the key and the base URL out of any error string before returning it. src/config.save writes .env at 0600 with os.open BEFORE the file exists, so it is never world-readable even for an instant. No credential has ever been in this repository.

The experimentWe did not attack it -- and the boundary that matters most on this vertical was proven by breaking it on purpose

An indirect prompt injection needs a field an outside party controls that reaches the prompt, and this corpus has none: 24 generated sentences, one generator, one seed. So instead of an attack trial the kit red-proves the boundary it CAN prove. evals/check_labels.py --self-test empties src/select.NEVER_SENT and requires the privacy check to convict; it leaks a week count and an open-item line onto a sheet and requires the leak scan to convict; it corrupts a gold label and flattens the memory flag and requires the replay check to convict. Four of four. The first version of that self-test scored three of four, and the one that failed was the privacy check -- because it tested membership of the very tuple the test empties, so emptying it made the check vacuously pass while the section sailed into every prompt. The red-proof found the checker, not the corpus. Confirmed by assertion and by reading the recorded run, not by an attack trial, on 2026-08-23 -- no attack run was fired; see redteam.why.

Each boundary above was checked by running an assertion or by reading the recorded run, not by an attack trial. The privacy boundary was red-proven by emptying the guard and watching the check convict -- and that red-proof caught the check itself being tautological, which is the whole reason to red-prove one.

The result0 attack trials, five boundaries checked -- and the privacy boundary red-proven by removing the guard and watching the check convict, which is also how the check's own tautology was found.

The Route Notes ARE the field an outside party would influence in a real deployment and this kit sends them, deliberately, rather than hiding its own injection surface. On the shipped corpus they are drawn from 24 generated sentences, so there is nothing adversarial to measure; what that surface does under attack is unmeasured and is the first thing to fire if this is pointed at a real book.

HonestyWhat this does not prove

• Whether a real deployment's Route Notes -- prose a dispatcher writes freely, and on a commercial book prose a customer can dictate -- would carry an instruction the model follows. Not applicable to the shipped corpus, and the first thing to attack if this is pointed at a real one.
• Whether the provider retains prompts, and for how long. Outside the kit and unmeasured.
• Whether a forker's own sheets carry personal data in a section this kit DOES send. The guard names one section; a real sheet might put a contact name in the Route Notes.
• What the model does with a sheet whose sections are missing or reordered. evals/check_labels.py asserts all seven are present in this corpus, so the degraded path is guarded rather than measured.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

No dispute, no credit and no re-norm, non-configurable. This kit produces a verdict, a consecutive-week count, a norm status and a class code, and stops. Nothing in src/, evals/, tools/ or ui/ files a dispute with a facility, issues a credit memo, adjusts an invoice or writes a route's norm into a book of record.

Everywhere and nowhere -- it is a property of what is ABSENT. The only writers in the kit are evals/run.py (results/*.json), tools/build_corpus.py (data/) and src/state.save (data/state.json). There is no HTTP client anywhere except src/adapters, which speaks to one completions endpoint.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• IT IS NOT A CHECK ON WHETHER THE ANSWER IS RIGHT. The carried state is correct whatever the model says, which means a wrong verdict is reported wrongly for that week and nothing catches it -- see Eval.taxonomy's norm_never_unshifts row, where five weeks recommend a rebase that is no longer warranted and the state machine is entirely happy.
• IT IS NOT AN APPROVAL WORKFLOW. There is no reviewer, no queue and no sign-off. The output is a watchlist row; who reads it and what they do next is outside the kit.
• IT IS NOT A SCHEDULER. The kit is invoked, not woken. Nothing here fires the weekly run, nothing notices if it does not happen, and nothing alerts on a missed week -- which is a real gap on a monitor whose whole value proposition is a cadence. See Architecture.breaks_at_scale.
• IT IS NOT A DEFENCE AGAINST A POISONED SHEET. If the Open Items log falsely records an anomaly CLOSED, the streak resets, the open item clears, and the kit reports WATCH on a route that should be ALREADY_OPEN. Nothing detects it and there is no second source.
• IT IS NOT A LIMIT ON SPEND BEYOND CALL COUNT. src/budget.py counts calls, not dollars, because the kit does not know the rate card a forker is pointed at.

WatchedWhat is watched, and why that one

The kit measures 37 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

16 measured by the latest run 21 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (already_open_cells 32, documents 150, memory_cells 64, routes 50, stateless False, watchlist_cells 77, weeks_scored 150) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

8 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

triage · with the model in the path — 7 runs. Columns here are only ever compared with each other.

not a time series No two of these 7 runs measured the same system — they differ on already_open_cells, baseline, documents, max_tokens, memory_cells, provider, routes, stateless, thinking, watchlist_cells, weeks_scored, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 16 chips that all say so.

DeviationsWhat deviated

0 breaches across 8 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-run evals/check_labels.py (free) on any change to tools/build_corpus.py, src/norms.py, src/segment.py or src/select.py -- it is eight checks over 150 documents and takes under a second. Re-run the scored eval (150 calls) only when the prompt, the model or the corpus changes, and re-run the stateless control WITH it: the headline on this page is a difference between two arms, and half a difference is not a number. The two free floors cost nothing and should be re-run with every corpus change, because they are what tells you whether the corpus got easier. ⚠︎ AND THE KIT'S OWN CADENCE IS NOT THIS ONE. The thing being described above is how often to re-MEASURE. How often the monitor RUNS is weekly, because the unit is a service week and the facility's correction window is 30 days from the tip date -- a week reviewed after that shuts becomes a credit memo against an invoice the customer already has. Nothing in this kit schedules that run or notices a missed one.

In one lineNo framework, deliberately

No framework and no dependencies at all -- json, os, re, sys, time, random, datetime, argparse, collections, concurrent.futures, urllib, http.client and http.server, every one of them in the standard library. requirements.txt names nothing and says why.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph, and the one place it looks like one is a straight line. Each route is a chain of three weekly runs; each run is parse, assemble, call, parse, score, advance. There is no branching, no tool selection and no retry-with-a-different-prompt. Drawing it as a graph would suggest choices the code does not make.

The other sideWhat a framework costs you

• No scheduler. This is a monitor that carries state and is still INVOKED, not woken -- the cadence, the retry policy and the question of what to do when a week does not arrive are all outside the kit, and the last of those has a 30-day deadline behind it.
• No state store. src/state.py writes one JSON file atomically. Two runs against the same file would race, and nothing here prevents it.
• No observability beyond a result file. There is no trace, no span and no dashboard; the run file records every call's tokens, latency, finish reason and carried state, and that is the whole of it.
• No queue and no backpressure. evals/run.py runs 12 route chains concurrently by default and a provider under load is handled by a bounded backoff inside src/adapters, not by a rate limiter.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-ticket-anomaly on the same tier, memory removed (THE CONTROL), 2026-08-23. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 6 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b000-ticket-anomaly-deskband, b001-ticket-anomaly-allclear recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the population is re-read whole on each scheduled run, and the prior run's carried state is what makes a reading a CHANGE.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved; every route, ticket, tonnage, facility, customer, address and recorded reason is invented by the generator. See kits/UC0088-ticket-anomaly/data/SOURCES.md. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe verdict, the week count, the norm status and the anomaly class, per week, exact match against the computed answer key

For each of the 150 weeks and each of the four answered fields, did the reply equal the computed answer key? Verdict, norm status and class are compared case-insensitively as strings; the week count is compared as an integer, and a value that is not a whole number is scored absent rather than coerced to 0 -- 0 is a meaningful answer here.

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model. The same function both free floors and the stateless control are scored through.

Every grader on these pages scored the same 150 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 150 per field. A week whose reply did not parse counts as a MISS in every field, never as an exclusion.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, computed by src/norms.replay over the planted opening state and observations at generation time. This grader IS the reference, so its own TPR and TNR are not measurable.

How tight can the band be? No tuned threshold anywhere on the model path -- the reply decides. The denominators are printed beside every rate because three are small: 3 NOT_MONITORED cells, 9 flip cells, 15 closed_reset cells.

Cadence: Re-run evals/check_labels.py (free) on any change to tools/build_corpus.py, src/norms.py, src/segment.py or src/select.py. Re-run the scored eval (150 calls) only when the prompt, the model or the corpus changes -- and re-run the stateless control WITH it, because the headline on this page is a difference between the two and half a difference is not a number.

The decisionWhen to reach for it

In one lineDuplicate disputes: weeks whose correct answer is ALREADY_OPEN that were answered RAISE

Of the 32 weeks where an anomaly raised by an earlier run is still open, how many were raised again? Every one is a second dispute against tickets that already carry one, which the facility rejects and somebody then reworks by hand.

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model.

Every grader on these pages scored the same 150 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

duplicates / 32. It is a COST WITH A NAME, not a slice of an accuracy figure, which is why it is counted on its own.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl. A week is ALREADY_OPEN when the carried state says an item is open and the week is out of band -- both facts computed by src/norms.step, neither on any sheet.

How tight can the band be? No threshold. The verdict string decides.

Cadence: With every scored run, and with the stateless control beside it.

The decisionWhen to reach for it

In one lineThe cells whose answer cannot be derived from the week in front of you

On the 64 of 150 cells where running the week from a BLANK state gives a different answer from running it from its real carried state, how often was the verdict and the week count right?

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model.

Every grader on these pages scored the same 150 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

hits / 64, separately for verdict and for the week count. The subset itself is computed by src/norms.replay running every week twice, not labelled by hand.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/norms.replay, run twice per week -- once from the real carried state, once from the blank state a one-sheet reader would have.

How tight can the band be? No threshold.

Cadence: With every scored run.

The decisionWhen to reach for it

In one lineWatchlist recall and false alarms, counted apart and never summed

Of the 77 weeks a clerk must touch, how many were reported as something to touch? And of the 73 quiet weeks, how many were put on the watchlist anyway?

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model.

Every grader on these pages scored the same 150 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

recall = hits / 77 on the non-quiet weeks; false alarm rate = quiet weeks reported non-quiet / 73.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl. Quiet is CLEAR or NOT_MONITORED; everything else is a week somebody has to look at.

How tight can the band be? No threshold.

Cadence: With every scored run.

The decisionWhen to reach for it

In one lineThe tonnage this run would put in dispute with the facility

Add up the week totals of every route reported RAISE. Is the portfolio number the disposal desk would take to the facility on Monday right?

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model.

Every grader on these pages scored the same 150 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

|reported - true| / true, over the 150 weeks. Only RAISE counts: ALREADY_OPEN tonnage is already with the facility under an existing dispute and counting it again would double the number rule 4 exists to stop.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl week totals, summed over the RAISE weeks.

How tight can the band be? No threshold.

Cadence: With every scored run.

The decisionWhen to reach for it