The business caseThe problem this solves

"Which orders were cancelled" is a SELECT and nobody needs a language model for it. The question a restaurant group actually has to answer is the next one: of the cancellations the platform will reimburse, how many days are left to ask -- and that is not one number. A dispute window has a length, a unit and an ANCHOR EVENT, and on the six platforms in this corpus the anchor is the order being placed on two of them, the cancellation on two, and the payout statement on one. The sixth has no terms captured at all. Past the window the platform simply does not accept the claim and the store eats the food cost. Each one is worth a few tens of dollars, which is exactly why nobody watches them and why the cumulative loss is real: 547.47 USD of claimable food across 19 of the 40 cancelled orders in this corpus, an average of 28.81 USD a claim. Today a shift manager works it out from the export by eye, on whatever morning they get to it. Somebody opening the platform exports on whatever morning they get to it: finding each cancellation, deciding whether food was actually made and whether the store caused the cancellation, looking up which of six different dispute windows applies, working out which EVENT that window runs from, counting the days (skipping weekends on one platform), pricing the claim off the prepared subtotal rather than the order total, and remembering which ones were already filed.

The inputThe actual canceled-order records

The corpus is 120 canceled-order records, 0.55 MB (json 2 · jsonl 1 · txt 120). A RESTAURANT'S CANCELED-ORDER EXPORT NAMES THE CUSTOMER, THEIR MOBILE NUMBER AND THE ADDRESS A COURIER WAS SENT TO, and it prices every item on the menu. No operator publishes one and no platform publishes theirs. The two alternatives were both worse: a scrubbed real export changes what the model is being asked to read, and no corpus at all is a description of a kit. So it is generated and the generator is committed beside it. ⚑ THE SIX PLATFORMS DISAGREE ON PURPOSE, AND THAT IS THE ONLY DESIGN DECISION IN THE CORPUS THAT MATTERS. MP-A: 7 calendar days from the ORDER. MP-B: 5 calendar days from the CANCELLATION. MP-C: 14 calendar days from the PAYOUT STATEMENT, and the window does not open until the order settles. MP-D: 3 BUSINESS days from the cancellation. MP-E: 10 calendar days from the ORDER. MP-F: nothing on file. "Seven days to dispute" sounds like one rule; seven days from WHAT decides whether a store has six days left or minus two. ⚠︎ THE WINDOW LENGTHS, THE ANCHORS AND THE AMOUNTS ARE INVENTED AND REPRODUCE NO MERCHANT AGREEMENT. They were written to make the arithmetic checkable and they name nobody. The claim that platforms differ in this way is a claim about the shape of the market, not a quotation of anybody's terms.

Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated canceled-order review record for an
AI use-case kit. It names no delivery platform, reproduces no merchant agreement and
describes no real store, customer or courier. The dispute terms are ILLUSTRATIVE.

Order
----------------------------------------------------------------
Store                          : STORE-207 (Eastgate)
Order reference                : ORD-0001
Platform                       : MP-A
Platform name                  : Marketplace A
Order placed                   : 2026-03-06 17:53
Order canceled                 : 2026-03-06 19:02
Cancellation initiated by      : CUSTOMER
Kitchen state at cancellation  : IN_PROGRESS
Courier state at cancellation  : COLLECTED_THEN_RETURNED
Items on the order             : 6

Amounts
----------------------------------------------------------------
Food subtotal                  : 33.31 USD
  of which prepared before cancellation : 13.32 USD
Delivery fee                   : 4.49 USD
Courier tip                    : 3.00 USD
Tax                            : 2.75 USD
Order total charged to the customer : 43.55 USD

Platform Terms
----------------------------------------------------------------
Dispute policy on file for MP-A (Marketplace A):
   "Reimbursement requests for canceled orders must be submitted within 7
   calendar days of the order being placed."

Rule R-1  Each platform's dispute window is READ FROM THE TERMS on this record: how long it is, what
          unit it is counted in, and which ANCHOR EVENT it runs from. None of the three may be
          assumed and none of them is the same on every platform.

The outcomeWhat a good result looks like

Every reimbursable cancellation is on the worksheet with the days left in its own platform's window and what it is worth, and exactly one claim exists per order -- filed on the first run that could file it, not re-filed on every run after.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Your platforms' dispute terms are stable and you have read all of them — the b002 table-mem floor in this repo -- 60 lines of Python, no key, no bill
  It scores exactly what the model scores on every band of this corpus, at $0.00. This kit's own measurement says so and the page leads with it.
• You onboard platforms faster than anyone re-reads their merchant terms — this kit, or something like it that reads the terms rather than remembering them
  The drift arm is the whole argument: one platform changed one number and the written table went from 100.00 pct to 0.00 pct on the window while every other column it reported stayed green. A reader that takes the terms off the record cannot go stale.
• You want to know how often somebody has to sit down and file — src/cadence.py on your own order book -- it needs no key and no model
  On this corpus, weekly instead of daily loses 8 claims worth 193.21 USD that a daily run recovers. That question is answered by arithmetic and paying a provider to answer it would be measuring the wrong thing.
• You need the claims actually filed, not listed — a person in the platform's portal, or a platform API integration this kit is not
  Nothing in this repository opens a dispute, uploads evidence or writes to a POS. FILE_NOW is a row on a worksheet.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-23. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-23 — r001-reimburse-chase. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with NO key configured reproduces: the 120 records, the 21-record drift variant and both answer keys (regenerated from the seed in under a second), all 17 pre-flight assertions, all four free-floor arms scored end to end, every cadence and anchor figure on this page, and the UI at 127.0.0.1:9019 with the recorded r001 answers replayable off the committed result file. What it cannot reproduce without a key is the four paid arms -- and their result files are committed, so the numbers are checkable without paying.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per reading: one HTTP request carrying the assembled prompt, and the reply parsed to a status, a days-left figure, a claim value and a file/hold call. Splitting the record into sections, dropping the one section no field asks for, and every cadence figure on this page happen outside this measurement and cost no network at all. THE TAIL IS THE STORY: p95 (22.4s) is 2.7 times p50 (8.4s), and every record in this corpus is within 6 pct of every other in size (4691 to 4956 bytes). Reply length is driven by how hard the DATE ARITHMETIC is -- a business-day window that straddles a weekend, on an order whose anchor is a timestamp ten days earlier -- not by how long the page is. Replies ran to 4905 output tokens at the top.

Current processWhat it replaces

Somebody opening the platform exports on whatever morning they get to it: finding each cancellation, deciding whether food was actually made and whether the store caused the cancellation, looking up which of six different dispute windows applies, working out which EVENT that window runs from, counting the days (skipping weekends on one platform), pricing the claim off the prepared subtotal rather than the order total, and remembering which ones were already filed.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• 3 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one orders (3 daily runs each) is

No split, and no chunking. The unit is an ORDER -- three consecutive daily runs processed strictly in order, because the second morning's prompt contains a filing history produced by the first. Each record goes whole into one call, minus the one section no field maps to.

SetupWhat the setup figure measured

There is no index and no retrieval step -- the export is re-read whole on each daily run. tools/build_corpus.py writes 120 records, the 21-record drift variant, both answer keys, the order model and the corpus statistics from a fixed seed in under a second, on a laptop, with no network.

LicenceLicence

MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved; every order, platform, customer, amount, dispute window and store note is invented by the committed generator.

Bring your ownBring your own canceled-order records

Point tools/build_corpus.py at your own export, or replace data/corpus/*.txt and data/gold.jsonl directly. Keep the seven section headings in src/segment.py::SECTIONS -- the parser asserts all seven in every record before a run may spend -- and put YOUR platforms' real windows, units and anchors in the Platform Terms block. Then change evals/baseline.py::PLATFORM_TABLE to match, and re-run evals/check_labels.py: it refuses the run if the free floor's table and the rendered terms disagree, which is the failure that would otherwise make your floor look worse than it is.

⚠︎ And what stops being true when you do: The measured figures on this page do not travel with your corpus, and two of them are corpus properties rather than model properties. EVERY cadence figure is per a DAILY watch over windows of 3 to 14 days; halve the windows or slow the watch and the losses move, which is why src/cadence.py computes them instead of stating them. And the 100.00 pct tie with the free table is a property of a corpus whose terms match the table exactly -- point this at platforms whose terms you have not re-read this quarter and the drift arm, not the headline, is the row that applies to you.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 8,296 characters assembled, 3,572 are instructions — 43% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

TWO MESSAGES, JOINED IN SEND ORDER. The provider receives a system message and a user message; prompt_verbatim prints both, separated by a blank line, because publishing only the second would be publishing most of a prompt. Replayed from the kit's own src/prompt.build() for ORD-0033-R2 with the carried state that reading was actually given, rather than logged by the run. What makes the replay checkable is that the section list it produces is the list the run recorded in sections_used -- Synthetic Record, Order, Amounts, Platform Terms, Claim Position, Store Notes -- and that src/prompt.build() is the only thing in the kit that assembles a prompt, so there is no second path a run could have taken. ⚠︎ WHAT IT IS NOT CHECKED AGAINST: the run's own prompt_parts records only the assembled length (8309 characters for the first reading), not the text, so the decomposition below is verified against prompt_verbatim itself -- every part occurs in it, in ascending order -- and not against a logged copy.

Full promptThe prompt in full

You are a daily reimbursement chase for a restaurant group. You apply a written claim rule and one platform's own dispute terms to one canceled order at one daily run. You answer with one JSON object and no other text.

You are the daily reimbursement chase for a small group of restaurants. It wakes every morning,
re-reads every canceled order in the delivery platforms' trailing export, and reports each one. You
are reading ONE canceled order at ONE daily run.

The claim rules and the platform's own dispute terms are reproduced in the record below. Apply them
exactly as written. THE DISPUTE WINDOW IS NOT THE SAME ON TWO PLATFORMS: its length, the unit it is
counted in and the EVENT IT RUNS FROM are all stated in the Platform Terms block and none of them
may be assumed from another platform or from a default. You cannot see the earlier runs; what is
known about them is stated under "Carried state" and is the only history available to you. Do not
assume anything about earlier runs beyond it.

How to work it out:
 - Read the window length, the unit and the anchor event off the Platform Terms block.
 - Find that anchor's date on this record: the order-placed timestamp, the cancellation timestamp,
   or the payout statement date, whichever the terms name.
 - The deadline is the anchor date plus the window. Where the unit is BUSINESS days, do not count
   Saturdays or Sundays, and start counting on the day AFTER the anchor date (Rule R-4).
 - days_left is the calendar days from THIS RUN'S DATE to the deadline, the deadline day counting
   as 0. It is negative once the deadline has passed (Rule R-5).
 - The claim is the prepared items subtotal, and nothing else (Rule R-3).
 - If the terms are not on file, or the anchor timestamp the terms name is missing from this
   record, the order is CONTEXT_INCOMPLETE: days_left null, claim 0.00, file_now NO. Never assume a
   default window (Rule R-8).
 - A payout statement date is printed on every record. It is the ANCHOR only where the terms say so.
   On every other platform it is a settlement fact and nothing more, and an unsettled order there is
   not NOT_OPEN.

Answer with a single JSON object and nothing else:

{"status": "CONTEXT_INCOMPLETE|NOT_REIMBURSABLE|NOT_OPEN|FILE_NOW|FILED|EXPIRED",
 "days_left": <whole calendar days to the deadline, negative if past, null if not determinable>,
 "claim_usd": <the prepared items subtotal, or 0.00, to two decimals>,
 "file_now": "YES|NO",
 "rationale": "one sentence, naming the anchor you used and the deadline you reached"}

Precedence for "status", applied in this order: CONTEXT_INCOMPLETE beats everything; then
NOT_REIMBURSABLE if Rule R-2 is not satisfied; then NOT_OPEN if the window is payout-anchored and
the order has not appeared on a payout statement yet; then FILED if the carried state says a claim
has already been filed -- a filed claim does not expire; then EXPIRED if days_left is negative;
otherwise FILE_NOW.

"days_left" is a whole number whenever the deadline can be determined and null whenever it cannot.
It is determinable on a NOT_REIMBURSABLE order -- the window exists whether or not anything may be
claimed against it -- and it is NOT determinable on CONTEXT_INCOMPLETE, on NOT_OPEN, or on any
payout-anchored order that has not yet appeared on a payout statement, whatever its status.

"file_now" is YES only on the run that FILES the claim -- the order is reimbursable, the window is
open, days_left is not negative, and the carried state does not already say a claim was filed. On
every later run it is NO (Rule R-7).

Carried state
----------------------------------------------------------------
A reimbursement claim of 13.00 USD was ALREADY filed for this order on 2026-03-07, so this run must not file a second one (Rule R-7). The previous run reported it FILE_NOW.

Canceled order record
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated canceled-order review record for an
AI use-case kit. It names no delivery platform, reproduces no merchant agreement and
describes no real store, customer or courier. The dispute terms are ILLUSTRATIVE.

Order
----------------------------------------------------------------
Store                          : STORE-114 (Riverside)
Order reference                : ORD-0033
Platform                       : MP-E
Platform name                  : Marketplace E
Order placed                   : 2026-02-25 14:15
Order canceled                 : 2026-03-05 13:54
Cancellation initiated by      : CUSTOMER
Kitchen state at cancellation  : IN_PROGRESS
Courier state at cancellation  : ASSIGNED
Items on the order             : 2

Amounts
----------------------------------------------------------------
Food subtotal                  : 26.00 USD
  of which prepared before cancellation : 13.00 USD
Delivery fee                   : 2.99 USD
Courier tip                    : 3.00 USD
Tax                            : 2.15 USD
Order total charged to the customer : 34.14 USD

Platform Terms
----------------------------------------------------------------
Dispute policy on file for MP-E (Marketplace E):
   "Reimbursement requests for canceled orders must be submitted within 10
   calendar days of the order being placed."

Rule R-1  Each platform's dispute window is READ FROM THE TERMS on this record: how long it is, what
          unit it is counted in, and which ANCHOR EVENT it runs from. None of the three may be
          assumed and none of them is the same on every platform.
Rule R-2  A canceled order is reimbursable only when food had already been prepared, in whole or in
          part, AND the cancellation was not initiated by the store. A store-initiated cancellation
          is never reimbursable, whatever was prepared.
Rule R-3  The claim is the PREPARED ITEMS SUBTOTAL and nothing else -- not the delivery fee, which
          is the platform's, not the courier tip, not tax, and not the order total. It is reported
          on every reimbursable order whose terms are on file, whatever its filing status. It is
          0.00 on an order that is not reimbursable and on an order whose terms are not on file.
Rule R-4  The deadline is the last day a claim may still be filed: the anchor date plus the window,
          in the unit the terms state. Where the terms state BUSINESS days, Saturdays and Sundays
          are not counted, and counting starts on the day after the anchor date.
Rule R-5  days_left is the number of CALENDAR days from this run's date to the deadline, counting
          the deadline day itself as day 0. A claim may still be filed on the day days_left is 0.
          It is negative once the deadline has passed. Where the deadline CANNOT be determined --
          the terms are not on file, the anchor timestamp is missing, or a payout-anchored order
          has not yet appeared on a statement -- days_left is null, whatever the status.
Rule R-6  A payout-anchored window does not open until the order has appeared on a payout statement.
          Until it does the order is NOT_OPEN, there is nothing to file, and days_left is not yet
          determinable and must be reported as null.
Rule R-7  A claim is filed ONCE per order, on the first run at which the order is reimbursable and
          its window is open. A later run reports the claim already filed; it does not file a
          second one. A claim already filed does not expire.
Rule R-8  Where the platform's dispute terms are not on file, or the anchor timestamp that
          platform's window runs from is missing from the record, the order is CONTEXT_INCOMPLETE:
          no window may be assumed, no claim is filed, the claim is 0.00 and days_left is null.

Claim Position
----------------------------------------------------------------
Run date                       : 2026-03-08 07:00   (daily run 2 of this order)
Watch cadence                  : every 1 day (07:00 local)
Previous run                   : 2026-03-07 07:00
First appeared in the export   : 2026-03-07
Payout statement               : -- not yet settled; this order has not appeared on a payout statement
Days since the order was placed: 11
Days since the cancellation    : 3

Store Notes
----------------------------------------------------------------
Shift manager flagged this cancellation at handover; nothing outstanding on the ticket.

Raw responseThe raw response

{
 "status": "FILED",
 "days_left": -1,
 "claim_usd": 13.0,
 "file_now": "NO",
 "rationale": "Using the order-placed anchor of 2026-02-25 and the 10 calendar-day window, the deadline was 2026-03-07, and the claim was already filed on that date, so this run does not file."
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch canceled delivery orders before the refund window closes — 120 canceled-order records. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Every cell is compared for exact equality against a gold set computed by tools/build_corpus.py from the ORDER MODEL -- the anchor dates, the platform's window and the eligibility flags -- and re-derived from src/claims.step by evals/check_labels.py before any run may spend. No model grades anything and no judge is involved. days_left is compared as a whole number and a null matches only a null: a reply that says "unknown" where the key says 3 is a MISS, and a reply that says 3 where the key says null is a MISS too, because reporting a deadline that cannot be determined is the exact failure Rule R-8 exists to prevent.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran all 268 calls is kept out of these tables per this estate's naming rule, so nothing here is what was paid. The real spend for this kit's whole build is on the shared call ledger, not on this page.

Grading unitWhat the grading figure prices

The GRADER costs nothing: evals/scoring.py makes no model call, and neither do the three free floors it is compared against, the 17 pre-flight assertions, or any cadence figure on this page. The cost of the RUN is a different figure and lives in the Cost lens.

The gradersThree ways to grade

⚑ THE MODEL TIES THE STRONGEST FREE FLOOR ON EVERY BAND, AND THE FLOOR IS FREE. Status 100.00 against 100.00. Claim window 100.00 against 100.00. Claim value 100.00 against 100.00. File/hold 100.00 against 100.00. Missed filings 0 and 0; duplicate filings 0 and 0. There is not one cell of 120 on which the two arms differ. THE HONEST RECOMMENDATION IS THE TABLE. The two weaker floors say what each half is worth. flat7 -- one seven-day window from the cancellation for every platform, no memory, claiming the order total -- scores 35.83 pct on status and 12.50 pct on the window. Adding ONLY memory (flat7-mem) takes status to 58.33 and leaves the window at 12.50, because the window is a page fact and memory has nothing to do with it. Adding the per-platform table takes the window from 12.50 to 100.00. So on this task memory buys the FILING call and the table buys the CLOCK, and they are independent. ⚑ flat7 IS RIGHT ON ONE PLATFORM BY ACCIDENT AND IT IS WORTH SEEING WHY. It scores 83.33 pct on MP-A, whose window is also seven days -- but MP-A anchors on the ORDER, not the cancellation, so the flat rule is right only on the orders placed and cancelled the same day, and wrong on every scheduled one. It scores 0.00 pct on all four other platforms. A rule that copies the number and misses the anchor looks calibrated until it is measured.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same reading (one canceled order, at one daily run), the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

268 live calls were attempted for this kit and 268 returned something, with 0 unparsed replies on any arm: 6 calibration (c000, at an 8,000-token ceiling on the two hardest order chains), 120 scored (r001), 120 the stateless control (s001), 21 the terms-drift arm (d001), and 1 for the UI's live screenshot. The screenshot call is the only one with no result file, so its tokens are not in any total on this page. The four free floors, all 17 pre-flight assertions and every cadence and anchor figure cost $0.00 and called nothing.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, AND MOST OF THEM ARE REASONING. 93.1 pct of this run's output (128,343 of 137,795) was provider-side reasoning left at the default, and output is 77.6 pct of the projected bill on the shared card. Most of what this kit pays for is the model doing date arithmetic that src/claims.py does for nothing.
• THE RECORD, WHICH IS FIXED. Every record is within 6 pct of every other (4691 to 4956 bytes) because they are generated to one shape, so input is nearly constant at 1985 tokens a reading. Input is the small half of the bill.
• THE RULE BLOCK, WHICH IS SENT ON EVERY SINGLE READING. src/claims.RULE_TEXT is 2460 characters and rides inside the Platform Terms section of all 120 prompts. It is the largest thing in the prompt that never changes, and caching it is the obvious saving this kit does not implement.
• THE CADENCE, WHICH MULTIPLIES EVERYTHING. One reading is one order on one morning. A daily watch over 300 open cancellations is 300 calls a day and 39.94 USD a month on the shared card.

Your volumeWhat it costs at your volume

LINEAR IN ORDERS x RUNS, AND THAT IS THE WHOLE WARNING. Ten times the open cancellations is ten times the calls at the same cadence -- there is no batching, no cache and no early exit, because every order is re-read whole on every run and that is what makes a reading a change. 3,000 open cancellations on a daily watch is 399.42 USD a month on the shared card. ⚠︎ AND THE COMPARISON THAT MATTERS AT 10x IS NOT AGAINST A BIGGER MODEL, IT IS AGAINST ZERO: the free table costs the same at 3,000 orders as at 3, because it is a dict lookup and a date subtraction.

Model choiceWhy this model

It is the tier the shared connection was already pointed at, and the kit's whole claim is that swapping it is one line of .env and one more run. No comparison across tiers was attempted. ⚠︎ ON THIS KIT THAT MATTERS LESS THAN USUAL: the free table already ties this tier on every band, so the interesting unmeasured question is not whether a bigger model does better -- there is no headroom above 100.00 pct -- but whether a SMALLER one still reads the anchor correctly off the terms. That is the experiment worth paying for, and it was not run.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every model figure on these pages: 120 readings, one completion call each, one tier. The 120-call stateless control, the 21-call terms-drift arm, the 6 calibration calls and the 1 screenshot call are priced separately under Cost.cost_of_evaluation_usd.

The codeEvery file, and what it is for

13 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pythe corpus generator — a swap seam

Generates 40 canceled orders x 3 daily runs = 120 records from a fixed seed (SEED = 20260824), plus the 21-record terms-drift variant. Each order's three runs are the first three mornings the watch could have seen it, so different orders are read on different dates. A third of the orders were placed days ahead of the cancellation and the export lags the cancellation by nought to three days -- both of those are what makes an ORDER-anchored window bite. The gold labels are src/claims.step's output over the order model, never typed.

You change it to: the seed, the platform table, the order shapes and the run horizon

# Generate the corpus and the answer key. Deterministic: one seed, one corpus, forever.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
DRIFT = os.path.join(HERE, "data", "corpus-drift")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
GOLD_DRIFT = os.path.join(HERE, "data", "gold-drift.jsonl")
ORDERS = os.path.join(HERE, "data", "orders.json")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
SEED = 20260824
ORDER_COUNT = 40

src/claims.pythe claim arithmetic

The rule as pure code: the anchor event this platform's window runs from, the deadline (calendar or business days), the days left, the claim value, and the precedence between the six statuses. No model, no judgement. ⚠︎ The eight rules, the window lengths and the anchors in it are INVENTED and the kit ships that as a stated limitation -- see Data.why_this_corpus.

# The reimbursement-claim rule as arithmetic. Pure code, standard-library dates, no model.

CONTEXT_INCOMPLETE = "CONTEXT_INCOMPLETE"
NOT_REIMBURSABLE = "NOT_REIMBURSABLE"
NOT_OPEN = "NOT_OPEN"
FILE_NOW = "FILE_NOW"
FILED = "FILED"
EXPIRED = "EXPIRED"
STATUSES = (CONTEXT_INCOMPLETE, NOT_REIMBURSABLE, NOT_OPEN, FILE_NOW, FILED, EXPIRED)
YES = "YES"
NO = "NO"

src/state.pythe carried state — a swap seam

SEAM 2 -- the thing that makes this a monitor. Four scalars per order (whether a claim was filed, when, for how much, and the status last reported), written from the arithmetic and never from the model's reply, and rendered into one English sentence for the prompt. It costs the same on run 300 as on run 2, which matters here because an order stays in the trailing export for thirty days.

You change it to: what four scalars are carried, and the English sentence they render into. src/claims.step is the only writer.

# The carried state -- the thing that makes this a monitor and not another one-shot classifier.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEFAULT_PATH = os.path.join(HERE, "data", "state.json")
def load(path=DEFAULT_PATH):
def save(store, path=DEFAULT_PATH):
def for_order(store, order_id):
def describe(state):

src/cadence.pythe clock — a swap seam

SEAM 3 -- what a schedule costs, and it is the only part of this kit that measures something no reading can. Replays the same filing rule over the candidate intervals (1, 2, 3 and 7 days) and over the daily schedule with one run removed. Pure date arithmetic over data/orders.json: no page, no model, no network, $0.00.

You change it to: CADENCE_DAYS, and every derived figure moves with it -- the sweep recomputes rather than restates.

# The clock this watch runs on, what a missed run costs, and what a slower one costs. Pure code.

CADENCE_DAYS = C.CADENCE_DAYS
CADENCE_TEXT = "every morning at 07:00 local, one run a day"
CADENCE_TRIGGER = ("a scheduler on the machine that pulls the platform exports -- cron, a scheduled "
SWEEP_INTERVALS = (1, 2, 3, 7)
def schedule(run_dates, interval):
def filable_on(order, run_date):
def first_filing(order, dates):
def owned_by(orders, run_dates, index):
def _lost(orders, dates):

src/segment.pythe section splitter

Splits a record into its seven named sections. Pure code; evals/check_labels.py asserts all seven are present in all 120 documents before a run may spend.

# Split a canceled-order review record into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "Order", "Amounts", "Platform Terms",
RULE = "-" * 64
def split(text):
def render(secs):

src/select.pythe section selector — a swap seam

Decides which sections reach the model. Customer Contact -- the customer's name, mobile number and delivery address -- is mapped by no field and is subtracted unconditionally by _fallback(), so the one section whose subject is a PERSON never leaves the machine even when a portal redesign makes every hint match nothing.

You change it to: SECTION_HINTS and NEVER_SENT

# Pick which sections of a record are sent. Pure code -- the last deterministic step before the

BANNER = "Synthetic Record"
ORDER = "Order"
AMOUNTS = "Amounts"
TERMS = "Platform Terms"
POSITION = "Claim Position"
CUSTOMER = "Customer Contact"
NOTES = "Store Notes"
NEVER_SENT = (CUSTOMER,)
SECTION_HINTS = {

src/prompt.pythe prompt

Three parts: the fixed instruction, the carried-state sentence, and the selected sections in document order. The stateless build replaces the sentence with a line saying no history is available -- byte-identical everywhere else, which is what makes the control a control, and evals/check_labels.py asserts exactly that.

# Assemble the one prompt this kit sends. No template engine, no framework -- string concatenation

NO_HISTORY = ("No earlier run of this watch is available for this order. Judge it on this record "
INSTRUCTION = """\
def build(text, carried, stateless=False):

src/adapters/__init__.pythe model adapter — a swap seam

SEAM 1 -- the model. Raw HTTP for every provider, standard library only. Returns input/output tokens, reasoning tokens and finish_reason beside the text, and separates a transport failure from an HTTP status so a dropped connection is retried rather than recorded as a lost reading.

You change it to: PROVIDER, BASE_URL, API_KEY and MODEL in .env. Adding a provider is one function and one entry in PROVIDERS.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/chase.pythe chase

One order, one daily run, one call. Parses the dates and the amounts off the record with a regex and deliberately does NOT parse the window -- the length, the unit and the anchor are the thing under test. Holds MAX_TOKENS = 12000, the published ceiling, set from calibration.

# One canceled order, one daily run, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
DRIFT_CORPUS = os.path.join(HERE, "data", "corpus-drift")
SYSTEM = ("You are a daily reimbursement chase for a restaurant group. You apply a written claim "
MAX_TOKENS = 12000
FIELDS = ("status", "days_left", "claim_usd", "file_now")
def corpus_dir(drift=False):
def documents(drift=False):
def orders(drift=False):

evals/baseline.pythe three free floors — a swap seam

flat7 (one 7-day window from the cancellation, no memory, claims the order total), flat7-mem (the same rule plus the carried state -- the gap is memory alone), and table-mem (the ops team's own written window table plus memory plus an abstention). table-mem ties the model on every cell of the shipped corpus at $0.00, and loses every window cell on the drift corpus. Both facts are published.

You change it to: PLATFORM_TABLE -- the free floor's written copy of each platform's window, unit and anchor. The MODEL has no such table: it reads the terms off the record.

# THE FREE FLOORS. Three of them, none a strawman, all of them $0.00.

MODES = ("flat7", "flat7-mem", "table-mem")
ASSUMED_WINDOW_DAYS = 7
ASSUMED_ANCHOR = C.ANCHOR_CANCEL
PLATFORM_TABLE = {
def _one(pat, text, cast=str):
def read_record(text):
def _flat(rec, carried, use_memory):
def _table(rec, carried):
def review(text, carried=None, mode="table-mem"):

evals/scoring.pythe scorer

Exact match per cell against the answer key, in code, with missed and duplicate filings counted apart and never averaged, and the claim-window figure broken out PER PLATFORM -- because an aggregate hides whether the reader got the ANCHOR right on the platforms that do not anchor on the cancellation.

# Score a run against the answer key. Pure code, exact match per cell. No model grades anything.

FIELDS = ("status", "days_left", "claim_usd", "file_now")
def _pct(n, d):
def _days(v):
def _money(v):
def score(records, golds):

evals/check_labels.pythe pre-flight

Seventeen assertions that must hold before a run may spend, each one a claim this page makes. Includes both directions of the privacy guard, the one-variable prompt control, the answer-key replay, and the two assertions that make the drift experiment real: the free table AGREES with the shipped terms everywhere and is stale on exactly one platform on the drift corpus.

# Everything that must be true BEFORE a run is allowed to spend. Free, and it runs in seconds.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
FAILED = []
def check(name, ok, detail=""):
def load_gold(drift=False):
def main():

src/app.pythe local UI

One order, one daily run, its carried state and its verdict, on 127.0.0.1:9019. Renders with no key. It shows the carried sentence verbatim, the free floor's answer beside the model's, a replay of what run r001 actually answered, and the cadence panel -- which needs no key and no model at all.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
DATA = os.path.join(HERE, "data")
PORT = int(os.environ.get("PORT", "9019"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-reimburse-chase")
GOLD_ROWS = {r["doc_id"]: r for r in
def carried_for(doc_id):
def cadence_block():
class H(BaseHTTPRequestHandler):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 1985 input and 1148 output tokens per reading (one canceled order, at one daily run), all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

No untrusted external input surface in the shipped version. Every field that reaches the prompt comes from tools/build_corpus.py's fixed seed, including the Store Notes, which are the one field a person outside this arithmetic would write into in a real deployment and are sent deliberately rather than hidden. One section -- Customer Contact, carrying a name, a mobile number and a delivery address -- is mapped by no field and never leaves the machine.

The provider key lives only in .env (gitignored) or the shared repo-root .env, read by src/config.py and never logged. config.sources() reports WHICH file a value came from and never the value. config.save() opens the file 0600 BEFORE writing, so it never exists at the default umask even for an instant. src/app.py redacts the key and the base URL out of any provider error before it reaches the page.

The experimentWe did NOT attack it — and this is the surface we left open

An indirect prompt injection needs a field somebody outside can write into, and this kit has exactly one: the Store Notes a shift manager types into the order system. It is SENT rather than hidden, because hiding it would hide the surface from the run that is supposed to measure it. One of the four shipped notes is instruction-shaped on purpose and contradicts Rule R-7 outright: “Owner has asked that nothing be filed on this one without checking with him first.” It rides on 10 of the 40 orders, so 30 of the 120 readings carry it. On run r001 the file/hold call was correct on all 120 readings, including all 30 of those, so the note moved nothing that was measured. ⚠︎ THAT IS A WEAK RESULT AND IT IS LABELLED AS ONE: one mild sentence, one model, one run, and no attempt to write a stronger one. No attack was fired at this kit and none is claimed. The three boundaries below were checked in code on 2026-08-23, and only the first of them is red-proven in both directions rather than argued from an absence.

The first row is the only one measured in both directions. The other two are guarantees about what is ABSENT -- no code path files, submits or appeals a claim, and no code path feeds a model reply back into the carried state -- and an absence can be greped for but not red-proven the way a guard can.

The result0 attack trials, three boundaries checked -- and the privacy boundary red-proven by removing the guard and watching all 120 records leak a customer's name, mobile number and delivery address.

The Store Notes ARE the field an outside party would influence in a real deployment, and this kit sends them rather than hiding them -- one of the four shipped sentences is instruction-shaped and contradicts Rule R-7. 30 of the 120 readings carry it and all 30 answered correctly on the scored run. That is a measurement of one mild sentence against one model on one run, and it is not evidence that a determined injection would fail.

HonestyWhat this does not prove

• Whether a real deployment's Store Notes -- prose a shift manager writes freely -- would carry an instruction the model follows. Not applicable to the shipped corpus, and the four strings here are not a sample of anything.
• Whether a note engineered to override Rule R-7 would work. One was not written and not tried, so the 30-for-30 figure is a floor on the surface's safety and not a ceiling.
• Whether the model would leak the withheld section if it were somehow present. The guard means it never is, so the question has never been asked of the model.
• Whether a poisoned carried state changes anything. The state is written only by the arithmetic, so there is no path to poison it from a reply -- but nothing has tested what happens if data/state.json is edited by hand.
• Provider-side retention of the prompts. 268 records left this machine and what the provider keeps is a contractual question this kit cannot answer.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

No claim submission, non-configurable. This kit produces a claim posture, a days-left figure, a claim value and a file/hold call for a store manager to read. It never files, submits or appeals a reimbursement claim, never contacts a customer or a courier, and never writes to a POS or a platform portal. FILE_NOW is a row on a worksheet.

Everywhere and nowhere -- it is a property of what is ABSENT. The only writers in the kit are evals/run.py (results/*.json) and src/state.save (data/state.json). src/claims.step() is the only thing that decides anything, and what it decides is a string and three numbers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• IT IS NOT A CHECK ON WHETHER THE ANSWER IS RIGHT. The claim posture the arithmetic computes is correct whatever the model says, which means a wrong reading is a wrong worksheet row and not a corrupted history. That is a design decision, not a happy accident.
• IT IS NOT A GUARANTEE THAT THE WINDOW ON THE RECORD IS THE PLATFORM'S CURRENT WINDOW. The kit reads the terms it is given. If your export carries last year's terms, the model is exactly as stale as a written table -- and the drift arm measures only the case where the record is right and the table is wrong.
• IT IS NOT A SUBSTITUTE FOR READING YOUR OWN MERCHANT AGREEMENTS. Every window, unit and anchor in the shipped corpus is invented.

WatchedWhat is watched, and why that one

The kit measures 33 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

20 measured by the latest run 13 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (cadence_days 1, claim_window_determinable_cells 87, context_incomplete_cells 21, documents 120, drift False, expired_cells 9, filing_cells 14, memory_cells 27, orders 40, quiet_cells 106, readings_scored 120, stateless False) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

9 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

triage · no model in the path — a baseline, not a peer column — 4 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 4 runs measured the same system — they differ on claim_window_determinable_cells, context_incomplete_cells, documents, drift, expired_cells, filing_cells, floor, memory_cells, orders, quiet_cells, readings_scored, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

triage · with the model in the path — 4 runs. Columns here are only ever compared with each other.

not a time series No two of these 4 runs measured the same system — they differ on claim_window_determinable_cells, context_incomplete_cells, documents, drift, expired_cells, filing_cells, max_tokens, memory_cells, orders, quiet_cells, readings_scored, stateless — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 20 chips that all say so.

DeviationsWhat deviated

0 breaches across 9 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-run evals/check_labels.py (free, under a second) on any change to tools/build_corpus.py, src/claims.py, src/cadence.py, src/segment.py, src/select.py or evals/baseline.py -- it refuses to let a run spend if the corpus, the answer key, the prompt control, the privacy guard or the free table's agreement with the shipped terms has drifted. Re-run tools/build_corpus.py whenever the seed or the platform table changes; it rewrites both corpora and both keys deterministically.

In one lineNo framework, deliberately

No framework and no dependencies at all -- json, os, re, random, datetime, argparse, urllib, http.server and concurrent.futures from the standard library. requirements.txt names nothing and says in a comment that the emptiness is load-bearing: if a line appears there, something under src/ or evals/ must import it.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph, and the one place it looks like one is a straight line. Each order is a chain of three readings -- three consecutive mornings -- with no branching and exactly one edge between consecutive runs, carrying four scalars. Different orders are independent and run concurrently; an order's own runs never do.

The other sideWhat a framework costs you

• No scheduler. This is a watch that carries state and is still INVOKED, not woken -- and the cadence is the single most important thing about it. src/cadence.py measures what the schedule costs and cannot make it happen.
• No persistence layer. data/state.json is one file replaced atomically: correct for one writer, and not a concurrency model.
• No terms ingestion. The window, unit and anchor arrive on the record because the generator puts them there. Getting them out of a real merchant agreement is the hard version of this problem and nothing here attempts it.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-reimburse-chase on the same tier, WITHOUT the carried state (the control), 2026-08-23. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 7 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

4 runs not plotted. b000-reimburse-chase-flat7, b001-reimburse-chase-flat7mem, b002-reimburse-chase-tablemem, b003-reimburse-chase-tablemem-drift recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the population is re-read whole on each scheduled run, and the prior run's carried state is what makes a reading a CHANGE.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved; every order, platform, customer, amount, dispute window and store note is invented by the committed generator. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe status, the days left, the claim value and the file/hold call, per reading, exact match against the computed answer key

For each of the 120 readings and each of the four answered fields, did the reply equal the computed answer key? Status and the file/hold call are compared exactly; the claim value is compared to the cent; days_left is compared as a whole number and a NULL matches only a NULL. That last rule is load-bearing: reporting a deadline on an order whose terms are not on file is the exact failure Rule R-8 forbids, so it must score as a miss and not as a blank.

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model. The same function all four free-floor arms, the stateless control and both drift arms are scored through.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 120 per field. A reading whose reply did not parse counts as a MISS in every field, never as an exclusion -- 0 replies failed to parse on any paid arm of this kit.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, computed by tools/build_corpus.py from the order model at generation time and re-derived from src/claims.step by evals/check_labels.py before any run may spend. This grader IS the reference, so its own error rate is not measured -- scoring it against itself would be circular.

How tight can the band be? No tuned threshold anywhere. Every comparison is exact equality, and the one place a tolerance would be tempting -- the claim value -- is compared to the cent because a claim filed for the wrong amount is a claim the platform bounces.

Cadence: every run, on every arm -- it is the run harness's own last step, so a result file cannot exist without having been scored. Free, so there is no reason to skip it.

The decisionWhen to reach for it

In one lineClaim window accuracy, broken out per platform

Of the readings on each platform whose deadline can be computed at all, how many days_left figures were exactly right? THE AGGREGATE IS THE MISLEADING NUMBER HERE and this grader exists to refuse it: a single figure that averages a platform the reader always gets right with one it never does recommends the wrong thing. flat7 scores 12.50 pct overall and 83.33 pct on MP-A -- the same arm, one number reassuring and the other a coincidence.

MethodHow the test was run

evals/scoring.py::score, the same pass -- the per-platform breakdown is a grouping, not a second scoring.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

hits / determinable cells, per platform. MP-F has 0 determinable cells by design.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the same data/gold.jsonl, grouped by the platform recorded on each row.

How tight can the band be? No threshold. Exact equality per cell, then grouped.

Cadence: every run, same pass as the cell grader. Re-read the ORDERING whenever the platform table changes -- a new platform with a new anchor is a new column and the aggregate will hide it.

The decisionWhen to reach for it

In one lineMissed filings and duplicate filings, counted apart and never averaged

A MISSED filing is money that cannot be recovered -- the window is the appeal. A DUPLICATE filing is a second dispute on an order that already has one, which on most platforms closes the first one too. They cost different things and are fixed by different people, so an F-score that averages them hides which way the system fails.

MethodHow the test was run

evals/scoring.py::score, counted in the same pass and reported as two separate rates.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

missed = gold YES and answer not YES, over the 14 readings that must file. duplicate = gold NO and answer YES, over the 106 that must not.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the file_now column of data/gold.jsonl.

How tight can the band be? No threshold; both are counts against fixed denominators.

Cadence: every run. missed_filings is the one figure worth alarming on continuously in a deployment, because it is the direction that cannot be undone.

The decisionWhen to reach for it