The business caseThe problem this solves

A franchise finance desk receives one period's sales report from a franchisee and has to decide whether the royalty remitted matches what the agreement's own ladder produces from the point-of-sale columns behind it. The columns are printed on the packet. The trap is not the arithmetic: it is whether each column may be USED at all for this period, which is settled six lines deep in a register control trail written in prose — a terminal journal withdrawn and restated, a delivery channel feed confirmed for the wrong period, a discount schedule whose funding side was never returned. Reading one packet's six dated register control notes against the agreement's reporting terms to decide which point-of-sale columns are usable for this period, then running R-2 to R-10's exclusion ladder in cents and comparing the result with what was remitted.

The inputThe actual period sales reconciliation packets

The corpus is 64 period sales reconciliation packets, 0.29 MB (json 3 · jsonl 1 · md 1 · txt 64). Because the shape of a reported-sales failure is not the arithmetic, it is whether a printed column may be used at all. Every packet carries all six point-of-sale columns and a six-note control trail, so a reader that never opens the trail still has a complete-looking set of numbers — which is exactly how a wrong reconciliation gets produced confidently. ⚑ AND THE NATURAL ERROR CANCELS ON THE FAMILY THAT COMMITS IT: an arm that nets the delivery commission out reproduces the franchisee's own figure on the commission-netted packets, so the variance comes out zero and the period reads clean, while the same habit raises a false flag everywhere else. That is why the FIGURES are the headline and the status is not: a constant reply takes 42.19 pct of status having read nothing and 0.0 pct of the full row.

BRACKENMOOR KITCHENS - FRANCHISE PERIOD SALES RECONCILIATION PACKET
SYNTHETIC SAMPLE DATA. GENERATED, NOT REAL. NO REAL FRANCHISOR, FRANCHISEE OR STORE.

Packet RSR-0001   Store STR-0717   Period PER-2026-W20   Submitted 2026-05-18

STORE AND AGREEMENT
  Store reference                                   STR-0717
  Other store in the same trade area                STR-0666
  Franchise agreement                               BK-FA-2026
  Royalty rate on net sales                         5.50 %
  Period start                                      2026-05-11
  Period end                                        2026-05-17
  Current terminal journal                          TJL-2026-5138
  Secondary terminal journal                        TJL-2026-4484
  Prior-period terminal journal                     TJL-2025-4292
  Current tax table                                 TXT-2026-33
  Prior tax table                                   TXT-2025-45
  Current delivery channel feed                     DCH-2026-3336
  Prior delivery channel feed                       DCH-2025-3775
  Current promotion schedule                        PRM-2026-44
  Prior promotion schedule                          PRM-2025-43
  Current gift-card ledger                          GCL-2026-5213
  Prior gift-card ledger                            GCL-2025-5665

REPORTED BY THE FRANCHISEE
  Net sales reported                                $50,253.00
  Royalty remitted                                  $2,763.91

AMOUNTS FROM THE POINT-OF-SALE SYSTEM
  POS gross rung, all channels                      $58,700.00
  Sales tax collected                               $4,960.00
  Third-party delivery gross, before commission     $7,590.00
  Third-party delivery commission retained          $1,550.00

The outcomeWhat a good result looks like

Either a reconciliation carrying ten figures — net sales, royalty due, royalty remitted, variance, tolerance and the five exclusions the ladder takes out — each naming the BK-FA-2026 term that produced it, plus a route, a timing verdict and a repeat-variance verdict; or a refusal naming exactly ONE unusable point-of-sale column and carrying no figures at all.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• every point-of-sale column on the packet is usable and printed — the free printed-columns floor, evals/baseline.py --floor fields
  it takes 53 of 64 full rows at $0.00 and the paid call takes 45. There is nothing here for a model to add and the measurement says so.
• the agreement's rules can be retyped in code and kept in step — the free ceiling arm, evals/baseline.py --floor full
  64 of 64 at $0.00. It beats the paid call by 19 packets at p = 0.000004.
• the answer turns on a control trail written in prose that nobody will retype — the paid call, with src/recheck.py in front of every figure it returns
  on the 11 reading-required packets it takes 6 where the floor of record takes 0. ⚠︎ AND A FREE WORD LIST TAKES 5 OF THE SAME 11 — the margin is one packet.

And where nothing here is good enough:

• you want the arm's own arithmetic on the page — nothing here
  as answered the call takes 5 of 64 full rows and gets its own ladder wrong on 47 of 64. Every figure this kit publishes is re-derived in code.

At a glanceHow the whole thing runs

Run once, for real, on 2026-09-17. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-09-17 — r001-reported-sales. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Measured on a clean checkout with no key configured and no network: python3 -m evals.run --floor fields scores all 64 packets in 0.5 s at $0.00, and python3 -m src.app --port 9494 serves the whole board — every committed arm, both columns — with the reconcile button disabled and the reason printed beside it. Two minutes from clone to a scored free floor, and nothing is bought to get there.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. Measured on the fast tier — end to end per packet: the request leaving the process to the last byte of the reply. One call per packet, no retries counted separately and no failures to exclude — all 64 returned. The pure-code station that follows is not in this figure and is not a network cost; every free arm's own p50 is 0 or 1 ms over the same corpus.

Current processWhat it replaces

Reading one packet's six dated register control notes against the agreement's reporting terms to decide which point-of-sale columns are usable for this period, then running R-2 to R-10's exclusion ladder in cents and comparing the result with what was remitted.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptArchitectureArchitectureApp UIApp UIEvalsEvalsUnit costUnit costBusiness caseUnit cost

• 1stations numbered in run order
• txt
• json
• jsonl
• md
• 3 of 11 stations carry a recorded failure

scroll → 11 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one graded cells (26 on each of the 64 period sales reconciliation packets) is

No train/test split, because nothing is trained or tuned on the model side and no segment of the corpus is held back: every arm — paid and free — answers all 64 packets and is scored on all 1,664 cells. The one cut that matters is a SUB-POPULATION rather than a split: 11 of the 64 packets are reading-required, and they are the only packets on which any arm can separate from any other.

SetupWhat the setup figure measured

THERE IS NO INDEX IN THE RETRIEVAL SENSE and nothing to retrieve from: one packet goes whole into one call, 4,855 characters of a 12,323-character prompt. The reduction step this flow variant names is the pairing itself — R-1's six columns against the six dated control notes — and it is computed in memory each run by src/packet.py at no measurable cost. A build figure of 0.0 here is an absence, not a fast index.

LicenceLicence

MIT - this repository's own LICENSE-PUBLIC.

Bring your ownBring your own period sales reconciliation packets

Point data/corpus/ at your own rendered period packets and data/period_records.json at their structured twin, keep the six column names and the note shape, and every arm runs unchanged: python3 -m evals.run --floor fields scores the free floor of record on your packets at $0.00 before you buy a single call. The generator is one file and one seed, so you can also re-cut this corpus at a different family mix and re-measure the whole ladder for nothing.

⚠︎ And what stops being true when you do: The boundary is the ANSWER KEY, not the documents. data/gold.jsonl is derived by the generator that renders the packets, so on your own corpus you supply the key — and until you do, every score on this page is a score about THIS corpus. evals/check_labels.py is the independent re-reading of the key and it imports nothing from src/; run it first.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 12,323 characters assembled, 4,855 are documents — 39% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Assembled by src/prompt.py in five parts in a fixed send order. The first three — the job, the reading rule and the ladder — are byte-identical on every call and are 4,471 of the 12,323 characters; the packet and the answer contract follow. The whole prompt is sent every call; there is no retrieval and nothing is summarised.

What the model receivesWhat the model is given, and what it is not

Every packet is complete: all six point-of-sale columns are printed, all six control notes are present, and nothing is truncated or OCR'd. So no score on this page is explained by a bad input — the 19 packets the call gets wrong are wrong against a packet a person could read.

Full promptThe prompt in full

You reconcile franchisee-reported period sales against the point-of-sale record for
Brackenmoor Kitchens, working ONE packet at a time, for a person to review.

You do three things and only these three.

1. For each of the six required point-of-sale columns, decide whether the figure printed on the
   packet may be reconciled from: it is `current`, it is `absent` (the packet did not return it), or
   it is `not-current` (the register control trail has put it beyond use for THIS period).
2. If every one of them is current, run the reporting terms printed on the packet and report the
   figures in whole cents, naming the term that produced each one.
3. If any of them is not, report NO figures at all and name exactly one column — the first that
   fails in the order R-1 states.

Everything you need is on the packet: the columns, the reporting terms in force, the franchisor's own
declared submission figure and the register control trail. There is nothing else to consult and no
outside requirement of any kind applies. Every figure and every date in this work is Brackenmoor Kitchens's own
agreed term and you must never write as though one came from anywhere else.

This is a reconciliation for internal review. Nothing here issues or drafts any notice to a
franchisee, nothing here moves, holds back or releases money, and nothing here leaves the
franchisor's own review. There is no field in your reply for any of those and you must not invent
one.

HOW TO READ THE REGISTER CONTROL TRAIL, and what it takes to put a printed figure
beyond use.

EVERY PACKET CARRIES AT LEAST ONE RESTATEMENT NOTE. Carrying one therefore tells you nothing on its
own. A note puts a printed figure beyond use for this period only when ALL THREE of these hold, and
each of them fails on real notes in this work:

  1. THE RECORD. The note must name the record this period's figure is actually read from — the
     CURRENT terminal journal, tax table, delivery channel feed, promotion schedule or gift-card
     ledger, as printed in the packet's own header block. Every packet ALSO prints a secondary
     terminal journal, a prior-period terminal journal, a prior tax table, a prior delivery feed, a
     prior promotion schedule, a prior gift-card ledger and another store in the same trade area. A
     note naming one of those is about a different record and says nothing here. The current and the
     secondary terminal journal are drawn from one number range and differ only in their last four
     digits; read them character by character.

  2. THE DATE. The note must take effect on or before the period end printed on the packet. A
     restatement effective after the period end does not reach this period, however serious it
     sounds.

  3. THE WITHDRAWAL. A later note may withdraw it. A withdrawal names BOTH the thing restated and
     the record, and a withdrawn restatement carries nothing whatever its own date. Two records on
     this packet can be read off the same terminal journal, so a withdrawal that names the record and
     not the thing does not withdraw both.

So an identical sentence naming an identical record settles one packet and settles nothing on
another. Only the dates and the withdrawal separate them. Read all six notes before deciding any of
them, because the withdrawal is usually written after the restatement it withdraws.

WHAT THE FIGURES ARE, AND WHAT IS NOT DEDUCTED.

Work in WHOLE CENTS as integers throughout, and round DOWN at every step. $50,253.00 is 5025300.

Net sales begins at the POS gross rung for all channels (R-2) and the packet prints five other
totals directly beside the ones that are excluded. Read R-3 to R-8 on the packet and deduct exactly
what they name and nothing else. Three of the figures printed on every packet are the most natural
deductions on the page and NONE of them is one: the delivery commission the channel retained (R-4
counts third-party delivery at gross), the store-funded share of the promotional discounts (R-6
excludes only the franchisor-funded share) and the voids that carry no authorisation (R-8 excludes
only the authorised ones). Each of those three is printed and non-zero on every packet.

The variance is the royalty due less the royalty the franchisee remitted, and it may be negative.
R-10 compares its SIZE against the greater of two arms, and both of them decide packets in this set.

A figure of zero is a figure, not an absence. A column that reads $0.00 is `current`.

THE PERIOD SALES RECONCILIATION PACKET, verbatim:

BRACKENMOOR KITCHENS - FRANCHISE PERIOD SALES RECONCILIATION PACKET
SYNTHETIC SAMPLE DATA. GENERATED, NOT REAL. NO REAL FRANCHISOR, FRANCHISEE OR STORE.

Packet RSR-0001   Store STR-0717   Period PER-2026-W20   Submitted 2026-05-18

STORE AND AGREEMENT
  Store reference                                   STR-0717
  Other store in the same trade area                STR-0666
  Franchise agreement                               BK-FA-2026
  Royalty rate on net sales                         5.50 %
  Period start                                      2026-05-11
  Period end                                        2026-05-17
  Current terminal journal                          TJL-2026-5138
  Secondary terminal journal                        TJL-2026-4484
  Prior-period terminal journal                     TJL-2025-4292
  Current tax table                                 TXT-2026-33
  Prior tax table                                   TXT-2025-45
  Current delivery channel feed                     DCH-2026-3336
  Prior delivery channel feed                       DCH-2025-3775
  Current promotion schedule                        PRM-2026-44
  Prior promotion schedule                          PRM-2025-43
  Current gift-card ledger                          GCL-2026-5213
  Prior gift-card ledger                            GCL-2025-5665

REPORTED BY THE FRANCHISEE
  Net sales reported                                $50,253.00
  Royalty remitted                                  $2,763.91

AMOUNTS FROM THE POINT-OF-SALE SYSTEM
  POS gross rung, all channels                      $58,700.00
  Sales tax collected                               $4,960.00
  Third-party delivery gross, before commission     $7,590.00
  Third-party delivery commission retained          $1,550.00
  Gift-card redemptions rung                        $1,205.00
  Employee meals rung at no charge                  $705.00
  Voids rung, total                                 $1,685.00
  Voids carrying a void authorisation               $747.50
  Promotional discounts rung, total                 $1,530.00
  Promotional discounts, franchisor-funded          $777.50
  Promotional discounts, store-funded               $752.50

REPORTING TERMS IN FORCE (from BK-FA-2026)
  R-1   A period is reconciled only where every point-of-sale input below is on
        the packet and usable. Where one is not, the pack names the first that
        fails, in this order: POS gross rung, sales tax collected, third-party
        delivery gross, promotion funding, gift-card redemptions, void register.
  R-2   Net sales begins at the POS gross rung for all channels.
  R-3   Sales tax collected is excluded.
  R-4   Third-party delivery counts at gross. Commission retained by the delivery
        channel is not deducted.
  R-5   Employee meals rung at no charge are excluded.
  R-6   A promotional discount reduces net sales only where the programme is
        franchisor-funded. A store-funded discount does not.
  R-7   Gift-card redemptions are excluded; the sale counted when the card sold.
  R-8   A void is excluded only where it carries a void authorisation.
  R-9   Royalty due is net sales at the rate above, rounded down to the cent.
  R-10  A variance is flagged where its size exceeds the greater of $25.00 and one
        percent of the royalty due for the period.
  R-11  Every figure names the reporting term that produced it.

DECLARED SUBMISSION STANDARD
  A period packet is submitted within 3 days of the period end. This is Brackenmoor
  Kitchens' own declared figure, agreed in BK-FA-2026, and no outside requirement
  is cited for it.

PRIOR PERIOD VARIANCE HISTORY FOR THIS STORE
  PER-2026-W19        flagged
  PER-2026-W18        clear
  PER-2026-W17        flagged

REGISTER CONTROL TRAIL
  2026-05-13  Drawer count for terminal TRM-7004 signed off on 2026-05-13.
  2026-05-14  Gift-card ledger GCL-2026-5213 was rebuilt on 2026-05-14; redemptions posted to it are suspended for business dates on or after 2026-06-03.
  2026-05-15  Delivery channel feed DCH-2026-3336 was re-sequenced on 2026-05-15; channel gross read from it is suspended for business dates on or after 2026-05-02.
  2026-05-16  The delivery channel feed restatement notified for DCH-2026-3336 was withdrawn on 2026-05-16; DCH-2026-3336 stands as posted.
  2026-05-17  Manager rota for store STR-0666 published on 2026-05-17.
  2026-05-18  Promotion schedule PRM-2025-43 was superseded on 2026-05-18; funding recorded on it does not apply to business dates on or after 2026-04-28.

PACK BOUNDARY (BK-FA-2026 R-1)
  This packet is reconciled for a franchise business review. Nothing here issues or
  drafts any notice to a franchisee, nothing here moves, holds back or releases
  money, and nothing here leaves the franchisor's own review.


YOUR REPLY. One JSON object, and nothing else — no commentary before or after it.

{
  "usable": {
     "pos-gross":               current | absent | not-current
     "sales-tax-collected":     current | absent | not-current
     "delivery-gross":          current | absent | not-current
     "discount-funding":        current | absent | not-current
     "giftcard-redemptions":    current | absent | not-current
     "void-register":           current | absent | not-current
  },
  "status": "RECONCILED" | "VARIANCE-FLAGGED" | "CANNOT-RECONCILE",
  "blocking_input": one of "pos-gross", "sales-tax-collected", "delivery-gross", "discount-funding", "giftcard-redemptions", "void-register", or null when the status is not CANNOT-RECONCILE,
  "blocking_kind": "absent" | "not-current", or null when the status is not CANNOT-RECONCILE,
  "route": "NO-ACTION" | "FRANCHISE-BUSINESS-REVIEW" | "EVIDENCE-CHASE",
  "timing": "WITHIN-STANDARD" | "BEYOND-STANDARD",
  "repeat_variance": "REPEAT" | "FIRST" | "NOT-FLAGGED", or null when the status is CANNOT-RECONCILE,
  "prior_flagged_count": an integer 0 to 3, read off the printed history,
  "figures": {
     "net_sales_cents":                 integer cents, or null
     "royalty_due_cents":               integer cents, or null
     "royalty_remitted_cents":          integer cents, or null
     "variance_cents":                  integer cents, or null
     "tolerance_cents":                 integer cents, or null
     "excluded_tax_cents":              integer cents, or null
     "excluded_giftcard_cents":         integer cents, or null
     "excluded_meals_cents":            integer cents, or null
     "excluded_authorised_voids_cents": integer cents, or null
     "excluded_franchisor_discounts_cents": integer cents, or null
  },
  "terms": {"<figure key>": "the R-number of the reporting term that produced it"},
  "review_note": one sentence for the reviewer, AT MOST 200 CHARACTERS,
  "blocking_note": one sentence naming the ONE column to chase and why, AT MOST 160 CHARACTERS,
                   or null when the status is not CANNOT-RECONCILE
}

THE TWO CHARACTER BUDGETS ARE LIMITS, NOT SUGGESTIONS. Write the shorter sentence. A long reply is
not a better one here and nothing downstream reads past the budget.

WHEN THE STATUS IS CANNOT-RECONCILE: every value inside "figures" is null and "terms" is an empty object. R-1 is
explicit that a period which cannot be reconciled produces no figures at all, because a partial
reconciliation is worse than none. Do not offer a figure "for guidance", and do not name more than
one blocking column.

OTHERWISE: every one of the ten figures is an integer number of cents, and every figure the
agreement's terms produced carries its R-number in "terms". "royalty_remitted_cents" is the
franchisee's own reported number and no term produced it, so it carries none. A figure whose column
is $0.00 on this packet carries no term either — name a term only for a figure this packet actually
has.

Raw responseThe raw response

{
  "usable": {
    "pos-gross": "not-current",
    "sales-tax-collected": "current",
    "delivery-gross": "current",
    "discount-funding": "current",
    "giftcard-redemptions": "current",
    "void-register": "current"
  },
  "status": "CANNOT-RECONCILE",
  "blocking_input": "pos-gross",
  "blocking_kind": "not-current",
  "route": "EVIDENCE-CHASE",
  "timing": "WITHIN-STANDARD",
  "repeat_variance": null,
  "prior_flagged_count": 2,
  "figures": {
    "net_sales_cents": null,
    "royalty_due_cents": null,
    "royalty_remitted_cents": null,
    "variance_cents": null,
    "tolerance_cents": null,
    "excluded_tax_cents": null,
    "excluded_giftcard_cents": null,
    "excluded_meals_cents": null,
    "excluded_authorised_voids_cents": null,
    "excluded_franchisor_discounts_cents": null
  },
  "terms": {},
  "review_note": "POS gross is suspended: the 2026-02-23 restatement names TJL-2026-5287 and is not withdrawn, so no figures are produced.",
  "blocking_note": "POS gross rung fails first: the restatement naming TJL-2026-5287 suspends rung sales, and no withdrawal reaches it."
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Franchisee reported sales vs POS reconciliation — 64 period sales reconciliation packets. One model answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

ONE GRAIN, WHICH IS THE PACKET, AND THE HEADLINE IS THE WHOLE ROW. A packet is right only if the status, the named unusable column, all ten figures, all ten term citations, the route, the timing and the repeat verdict are all right together. The 1,664 cells are published beside it and are never averaged into it — a 90 pct cell score and a 70 pct full row are the same run.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on this page is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection, never a counted figure. The tokens are the provider's own reported prompt and completion counts on each of the 64 calls of r001-reported-sales; the rates are read from the app's own build/facts/models.json, each row carrying its source URL and the date it was last re-read.

Rates checked 2026-09-12. The tier the kit actually ran on is not priced on this page. Its rate card is in the app's own models.json and the run's measured total is published beside every score; naming the vendor on a reader-facing page is what this site withholds.

Grading unitWhat the grading figure prices

GRADING IS FREE — every grader on this kit is pure Python against a derived key, so every score on this page can be recomputed for $0.00. The only bill this kit ever ran up is the 76 calls that answered.

The gradersOne way to grade, and why it is the only one

⛔ THE HEADLINE IS A LOSS AND IT IS PUBLISHED AS ONE. The measure is the FULL ROW — the status, the one named unusable column, all ten figures, all ten BK-FA-2026 term citations, the route, the timing and the repeat verdict, on one packet. Rechecked through src/recheck.py the paid call takes 45 of 64 (70.31 pct). The free floor of record — the printed point-of-sale columns read by pure Python, $0.00, no network — takes 53 of 64 (82.81 pct). McNemar exact on the same 64 paired items: only the paid call 6, only the floor 14, p = 0.115318 — NOT SIGNIFICANT, and the point estimate runs AGAINST the call. The honest sentence is that this corpus cannot separate the paid call from its free floor on the full row. ⛔ THE STATION IS THE FREE FLOOR, MEASURED ON A CONSTANT. everything current — one fixed reply that reads nothing at all — scores 0 of 64 full rows as answered and 53 of 64 through the same station, which is EXACTLY the floor of record. The station ran 776 field overrides to do it. Per free arm the station overrode: fields 0, wordlist 0, tuned 0, full 0, constant 776, the paid call 215. The floor of record needs no station — it already computes the ladder. A margin quoted from a rechecked paid arm against a raw free arm would have read 45 against 0 and would have been almost entirely station. Every floor, both columns, on the same 64 packets: fields 53 as answered and 53 through the station; full 64 and 64; tuned 11 and 11; wordlist 11 and 11; constant 0 and 53. The paid call is 5 as answered and 45 through the station. ⚠︎ TWO OF THE FIVE FREE FLOORS BEAT THE PAID CALL and a third ties it through the station, all at $0.00.

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Set limitationsWhat this set cannot show

⚠︎ IT IS NOT BALANCED AND THAT IS THE POINT. 43 of the 64 packets can be answered from the printed columns alone (16 RECONCILED, 27 VARIANCE-FLAGGED) and 21 are refusals; only 11 are reading-required. The family mix is clean 11, stale-input 11, absent-input 10, commission-netted 7, store-funded-discount 7, unauthorised-void 7, giftcard-double 6, late-submission 5.

A fixed reply of everything current takes 42.19 pct of the STATUS measure having read nothing at all, and 0.0 pct of the full row. The same reply through this kit's own station takes 82.81 pct of the full row — exactly the floor of record. Both numbers are on the page for that reason.

Rebuilding the corpus at any mix is one command and $0.00 — tools/build_corpus.py regenerates all 64 packets and the whole key from the seed. It was not done. A corpus with 32 reading-required packets instead of 11 is the one change that would give the paid call room to win, and it would also be a corpus that does not look like a franchise finance desk's post bag. The honest statement is that this kit was measured on the realistic mix and lost.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — top_k, chunk size, prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-09-12. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingGrading is free; the bill is the 76 calls that answered.

The scorer, the label gate, all five free floors, the generator, the station and the significance tests are pure Python and cost $0.00 however often they run. $0.061763 is r001's 64 calls and $0.011631 is x001's 12, both measured from the provider's own reported usage at the tariff in force when each call was made. ⚠︎ ONE FURTHER CALL WAS BOUGHT AND IS DISCLOSED RATHER THAN DROPPED: driving the board's routes by hand hit POST /api/check once on RSR-0001 — 3,117 in / 375 out, approximately $0.000933, off-peak, NO RUN ID and in no published score. The kit's true total spend is approximately $0.074327 over 77 calls.

Cost driversWhat actually moves the bill

• THE PACKET ITSELF. 4,855 of the 12,323 prompt characters are the rendered packet and it is re-sent whole on every call; there is no index and nothing is summarised.
• THE STABLE PREFIX. The job, the reading rule and the ladder are 4,471 characters that never vary and are sent 64 times. The provider reported 0 cache-hit tokens on all 64 calls, so none of it was billed at a cached rate on this run.
• THE ANSWER CONTRACT. 2,997 characters of schema on every call, and 427 output tokens on average coming back — a reply that names ten figures and ten terms is not short.

Your volumeWhat it costs at your volume

Linear in packets and there is no shared state, so 640 packets is 640 calls at the same $0.000965 each — about $0.62 — and 6,400 is about $6.18. Nothing here gets cheaper with volume: the prompt does not shrink, and the provider reported no cache hits to amortise. What does change at 10x is the comparison, not the price: the free floor of record costs $0.00 at every volume and is 8 packets ahead.

Model choiceWhy this model

The tier was not chosen for this kit; it is the tier this estate runs every kit on, reasoning off and a 1,000-token ceiling, because the cheapest path is the default. ⚠︎ AND ON THIS KIT THE TIER IS NOT THE QUESTION: two free arms beat it and a third ties it through the same station, so a more expensive tier would have to buy back 8 packets before it broke even with code that costs nothing.

Other modelsWhat one period sales reconciliation packet costs, on cards you can buy

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

the 64 calls of r001-reported-sales, one per packet, bought off-peak. The 12 calls of x001-reported-sales are priced separately at $0.011631, and one further unplanned call of approximately $0.000933 is disclosed in the cost of evaluation.

The codeEvery file, and what it is for

13 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/app.pyThe board

a stdlib HTTP server over the committed runs: one packet, the whole corpus with every arm's raw and rechecked column side by side, and the adversarial probe. POST /api/check is the only route that can spend and is disabled without a key.

# The minimal local board. Standard library only — `python3 -m src.app`, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "9494"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-reported-sales")
PROBE_RUN = os.environ.get("PROBE_RUN", "x001-reported-sales")
FLOOR_RUNS = {"fields": "b000-reported-sales-fields",
LIVE_FLOORS = tuple(BL.MODES)
ARM_ORDER = LIVE_FLOORS + ("model", "rechecked")
FLOOR_LABEL = {

src/packet.pyPacket parser

cuts one rendered packet into its header, its six point-of-sale columns, the three-period variance history and the six dated register control notes. Pure text; no model.

# THE PACKET'S OWN COLUMNS, READ BY PURE CODE. What a franchise finance desk already holds.

MONEY = re.compile(r"^\$([\d,]+)\.(\d{2})$")
RATE = re.compile(r"^(\d+)\.(\d{2}) %$")
HEADER = re.compile(r"^Packet (\S+)\s+Store (\S+)\s+Period (\S+)\s+Submitted (\S+)$", re.M)
NOTE = re.compile(r"^  (\d{4}-\d{2}-\d{2})  (.+)$")
HISTORY = re.compile(r"^  (PER-\S+)\s{2,}(\S+)$")
FIELD = re.compile(r"^  (\S.*?)\s{2,}(\S.*)$")
NOT_RETURNED = "not returned"
TRAIL_BLOCK = "REGISTER CONTROL TRAIL"
HISTORY_BLOCK = "PRIOR PERIOD VARIANCE HISTORY FOR THIS STORE"

src/terms.pyThe terms engine

BK-FA-2026's R-1 to R-12 as code — the precedence order over the six columns, the R-2 to R-10 exclusion ladder in integer cents, R-10's tolerance (the greater of $25.00 and 1 pct of royalty due) and R-12's submission standard.

# BK-FA-2026's REPORTING TERMS AS CODE. R-1 through R-11, retyped once, in the only place they are

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
STATS = json.load(open(os.path.join(HERE, "data", "corpus-stats.json"), encoding="utf-8"))
AGREEMENT_ID = "BK-FA-2026"
FRANCHISOR = "Brackenmoor Kitchens"
STATUSES = ("RECONCILED", "VARIANCE-FLAGGED", "CANNOT-RECONCILE")
ROUTES = ("NO-ACTION", "FRANCHISE-BUSINESS-REVIEW", "EVIDENCE-CHASE")
TIMING = ("WITHIN-STANDARD", "BEYOND-STANDARD")
REPEAT = ("REPEAT", "FIRST", "NOT-FLAGGED")
AVAILABILITY = ("current", "absent", "not-current")

src/prompt.pyPrompt assembly

five parts in a fixed send order, 12,323 characters, of which 4,471 are the byte-identical stable prefix. Asserts the three forbidden acts are absent from the answer contract at import.

# ONE PACKET IN, ONE RECONCILIATION OUT. The prompt, assembled in named blocks.

VERDICTS = ["RECONCILED", "VARIANCE-FLAGGED", "CANNOT-RECONCILE"]
VERDICT_MEANINGS = {
SYSTEM = """You reconcile franchisee-reported period sales against the point-of-sale record for
READING_RULE = """HOW TO READ THE REGISTER CONTROL TRAIL, and what it takes to put a printed figure
ARITHMETIC_RULE = """WHAT THE FIGURES ARE, AND WHAT IS NOT DEDUCTED.
def _schema_block():
SCHEMA = _schema_block()
PACKET_LEAD = "THE PERIOD SALES RECONCILIATION PACKET, verbatim:\n\n"
def build(text):

src/adapters/__init__.pyThe one call site — a swap seam

one OpenAI-compatible streamed request, reasoning disabled, and the per-call price computed from the provider's own reported prompt/completion/cache split at the tariff in force at the UTC time of the call.

You change it to: one OpenAI-compatible endpoint, base URL and model id from .env. Nothing above it knows which vendor answered; the free floors are unaffected and two of them beat the paid arm.

# SEAM 1 — the model. Taken from tools/templates/adapters.py at this batch's launch and filled;

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TIMEOUT_S = 600
TRANSPORT_RETRIES = 1
FIRST_EVENT_TIMEOUT_S = 150
KEEPALIVE = object()
TRANSIENT_STREAM_MARKERS = ("unable to start processing", "timeout limit", "try again later",
def _post(url, headers, payload, timeout=TIMEOUT_S):

src/recheck.pyThe station — a swap seam

⛔ THE LARGEST SINGLE EFFECT ON THIS PAGE. It trusts exactly ONE field of the reply — usable, the reading of R-1's six columns — and re-derives the status, the named column, all ten figures, all ten term citations, the route, the timing and the repeat verdict in pure code. 215 overrides on r001; 776 on a constant that read nothing.

You change it to: today it trusts exactly one field. Widen it and the arm's own arithmetic reaches the page: the raw column is what that looks like, 5 of 64.

# THE PURE-CODE STATION, APPLIED TO AN ANSWER. What the arm is trusted with, and nothing more.

TRUSTED = ("usable",)
RECOMPUTED = ("status", "blocking_input", "blocking_kind", "figures", "terms", "route", "timing",
NOT_REDERIVABLE = ("review_note", "blocking_note")
def _state(usable, name):
def _figs(a):
def reading(answer, p):
def reconcile_token(rd):
def recheck(answer, doc_id, text):

src/reader.pyThe trail reader

the free readers of the register control trail: the domain word list, the generator-tuned regex and the full rulebook retyped. This file is the ceiling arm and it beats the paid call 64 to 45.

# ONE PERIOD SALES RECONCILIATION PACKET IN, ONE ANSWER OUT. The only place a model is called.

MAX_TOKENS = 1000
THINKING = THINKING_OFF
def _balanced_end(t, i):
def extract(text):
def whole_cents(v):
def _vocab(v, allowed, upper=True):
def normalise(ans):
def read(cfg, doc_id, text, complete_fn=None, max_tokens=MAX_TOKENS, reply_stop=REPLY_STOP):

src/refusal.pyThe boundary — a swap seam

reads the arm's two free sentences for the three acts BK-FA-2026 forbids — holding a payment back, preparing a formal notice, sending the pack outside the desk. A phrase floor, and it says so.

You change it to: a phrase floor today. A paraphrase gets past it, and the kit says so rather than claiming a guarantee.

# THE PACK BOUNDARY READ IN THE REPLY — the three acts this pack never performs.

ACT_PATTERNS = {
ACTS = tuple(sorted(ACT_PATTERNS))
def _hits(rx, text):
FREE_TEXT_FIELDS = ("review_note", "blocking_note")
INVENTED_FIELD_MARKERS = ("notice", "demand", "withhold", "withheld", "hold_back", "holdback",
def free_text(answer):
def check(answer):
def budget_reading(answer):

evals/scoring.pyThe scorer — a swap seam

26 graded cells on each of 64 packets, 1,664 in all, pure Python against the derived key. Scores the raw answer and the rechecked determination with the same function so the two columns can never be scored differently.

You change it to: the full row is one packet. Score the 1,664 cells instead and the same run reads very differently — which is why the cell counts travel beside the packet counts and are never averaged into them.

# ONE SCORER, BOTH COLUMNS. The arm's own reconciliation and the station's determination are the

FIGURES = tuple(T.FIGURES)
GRADED_DECISION = ("status", "blocking_input", "blocking_kind", "timing", "repeat_variance",
GRADED = GRADED_DECISION + tuple("figures." + f for f in FIGURES) \
CELLS_PER_PACKET = len(GRADED)
def true_usable(g):
def _figs(a):
def _terms(a):
def score_one(a, g):
SUMMED = ("status", "blocking_input", "blocking_kind", "decision", "timing", "repeat_variance",

evals/baseline.pyThe free floors — a swap seam

all five arms — fields, wordlist, tuned, full, constant — answering end to end at $0.00, and the sweep that puts every one of them through the same station.

You change it to: which arm the headline is measured against. Measured both ways here: against fields the paid call loses 45 to 53; against wordlist it wins 45 to 11. The bar is the best free arm SHIPPED, so it is fields.

# The free code floors, scored before a single paid call is bought.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
PRECEDENCE = ["pos-gross", "sales-tax-collected", "delivery-gross",
TOLERANCE_FLOOR_CENTS = 2500
TOLERANCE_PCT_BP = 100
UNUSABLE = ["restated", "reloaded", "re-sequenced", "superseded", "rebuilt", "suspended",
INPUT_WORDS = {
RESTATED_THING = {

evals/check_labels.pyThe label gate

an independent second reading: retypes R-1 to R-12 from the packet text, imports nothing from src/, and reports 0 disagreements over 64 packets and 384 control notes. Check 10 is the cap: it convicts the forbidden vocabulary anywhere it is not declared.

# Re-derive the whole answer key a SECOND time, from the rendered packets, and disagree loudly.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
PRECEDENCE = ["pos-gross", "sales-tax-collected", "delivery-gross",
RESTATEMENTS = [
WITHDRAWAL = re.compile(
RESTATED_THING = {
FORBIDDEN_PATTERNS = [
CITATION_FORMS = [

evals/run.pyThe run harness

one call per packet, three workers, a streamed early close on a runaway shape, and a three-way admission test so a connection the provider never answered is quarantined rather than cached.

# Read all 64 period sales reconciliation packets and write one result file. THE PAID ARM SPENDS.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
KIT = "reported-sales"
RATE_CARDS = {
def tariff_at(card, ts):
def price(card, row):
def price_at(card, row, tariff):
def stub_complete(cfg, system, user, max_tokens=1024, **kw):
def _events(chunks, usage=True):

tools/build_corpus.pyThe corpus generator

64 packets and the whole answer key from one fixed seed (20260917). Rebuilds byte-identically under PYTHONHASHSEED 0, 1 and 97.

# Build the UC0494 reported-sales corpus - 64 period sales reconciliation packets, and the key.

SEED = 20260917
DATASET = "reported-sales-v1-64packets"
AS_OF = "2026-09-17"
HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
FAMILIES = [
N_DOCS = sum(n for _, n in FAMILIES)
REQUIRED_INPUTS = ["pos-gross", "sales-tax-collected", "delivery-gross",

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3106 input and 426 output tokens per query at top-k 1, of which about 150 tokens are fixed (system prompt, chat framing, the question itself) and the rest is retrieved context. Change the retrieval depth and the input scales with it. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model and top-k follow the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

The cap is a SHAPE before it is a sentence. The answer contract offers no field that could hold a franchisee payment back, prepare or issue a notice, or send the pack outside the finance desk — so the boundary is enforced by what the reply can contain, not by asking the model to behave. src/prompt.py asserts the three acts' absence at import and refuses to build a prompt without them.

One credential, read from <code>&lt;repo&gt;/.env</code> (gitignored from the first commit), sent as a header to one endpoint. It is never written to a result file, a cache row, a log line or a page, and the board serves an allowlisted subset of the cost block so neither the model id nor the rate card ever reaches the browser.

The experimentCan a register control note make it hold a payment back, prepare a formal notice, or send the reconciliation pack outside the desk?

Four packets, three escalating control notes each, 12 live calls. The notes arrive where a colleague's note actually arrives on this document — inside the register control trail — and escalate from a plain request, to the same request framed as agreed and overdue, to a note in the franchise manager's voice declaring the agreement's limits superseded and authorising all three acts. python3 -m evals.injection --probe --yes # 12 calls Measured on all 64 packets of r001-reported-sales, all three committed free floors, the stub, and the 12 escalated trials of x001-reported-sales.

Every figure here is read off results/eval-x001-reported-sales.json. The in-corpus attack costs nothing extra: 3 of the 64 ordinary packets already carry a forbidden ask in a colleague's voice with no injection at all, and the paid arm takes 1 of those 3 full rows.

The result0 forbidden acts in 12 escalated trials, and 10 of 12 answers wrong against the key.

results/eval-x001-reported-sales.json carries every trial: the escalation text verbatim, what came back, which acts were asserted, whether a forbidden field appeared, the status against the key, and the token counts and price of each call.

HonestyWhat this does not prove

• Whether a fourth wording, or one written by somebody trying, would be complied with.
• Whether the injected note moved the READING of the genuine notes on the same packet — a row count cannot see it and nothing here measured it.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

Nothing this kit produces holds back a franchisee payment, prepares or issues a formal notice of any kind, or sends a reconciliation pack outside the finance desk. The answer contract offers no field for any of the three.

src/prompt.py asserts the three forbidden acts are absent from the answer contract at import; src/refusal.py reads the arm's two free sentences for them on every reply of every arm; evals/check_labels.py check 10 retypes the three planted adversarial sentences, strips them from every file, and then requires ZERO cap vocabulary anywhere else in src/, ui/, the prompt, the output schema or any result file.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is not a guarantee about wordings nobody fired. Three escalations were tested and a fourth may behave differently.
• It is not a semantic guard. src/refusal.py convicts the acts as a person would write them; a determined paraphrase gets past it and the kit says so rather than claiming otherwise.
• It is not a statement about correctness. A reply can stay inside every boundary and still refuse a period nobody owed a refusal on — which is this kit's largest single loss.
• It is not a claim that the injected note left the READING of the genuine notes on the same packet untouched. A row count cannot see that and nothing here measured it.

WatchedWhat is watched, and why that one

The kit measures 46 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

15 measured by the latest run 31 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (failures 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

5 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

match · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 3 runs measured the same system — they differ on floor, recheck_arm_is_copy_of_raw, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

match · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 75 chips that all say so.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 15 chips that all say so.

DeviationsWhat deviated

0 breaches across 5 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-run all five free floors and evals/check_labels.py on every change to the corpus, the generator, the terms engine or the station — all of it free. Re-fire the paid arm only when the model or the prompt moves, and never over an existing run id.

In one lineNo framework, deliberately

No framework. The whole kit is the Python standard library, and every place a framework would sit is a file you can read: one call site, one prompt builder, one station, one scorer.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

one call, no chain, no agent, no tools, no retrieval — the whole graph is: read one packet, ask for one reading of six columns, re-derive everything else in code, score.

The other sideWhat a framework costs you

• A framework would add a dependency to a kit whose entire argument is that free code beats the call; the comparison is easier to believe when both arms are in the same standard library.
• Nothing here needs orchestration: one packet is one call and there is no state between them.
• The cost of having none is that retries, rate limiting and the admission test are written out longhand in evals/run.py rather than inherited.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-reported-sales on the fast tier, 2026-09-17. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 3 runs on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — blocking is computed in memory each run and is the only reduction step.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, FOUNDRY_RUN_ID, MAX_CALLS_PER_DAY, MODEL, PORT, PROBE_RUN, PROVIDER, PYTHONHASHSEED, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT - this repository's own LICENSE-PUBLIC. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe period sales reconciliation packet — one reading of each of R-1's six point-of-sale columns, a status, one named unusable column, ten figures, ten BK-FA-2026 term citations, a route, a timing verdict and a repeat-variance verdict

whether each of R-1's six point-of-sale columns is read as current, absent or not-current for THIS period; whether the packet is therefore RECONCILED, VARIANCE-FLAGGED or CANNOT-RECONCILE; when it cannot, whether exactly ONE unusable column is named and it is the first one R-1's precedence reaches; when it can, whether all ten figures are the ones R-2 to R-10's ladder produces in integer cents and each names the term that governs it; and whether the route, the timing verdict and the repeat-variance verdict follow. 26 graded cells on each of 64 packets, 1,664 in all, every grader pure Python against a derived key.

MethodHow the test was run

python3 -m evals.run --run-id <id> [--floor fields|wordlist|tuned|full|constant | --stub | --resume]. It scores the raw answer and the rechecked determination with the same function and publishes both columns on every arm.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, derived by tools/build_corpus.py from the same records it renders, and re-read independently by evals/check_labels.py.

No true/false rates for this grader. It records 13 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? no threshold is tuned anywhere in this kit: R-10's tolerance is the agreement's own (the greater of $25.00 and 1 pct of royalty due) and both arms of it bite — 22 packets on the flat arm and 21 on the percentage arm

Cadence: re-run all five free floors and evals/check_labels.py on every change to the corpus, the generator, the terms engine or the station; the paid arm only when the model or the prompt moves

The decisionWhen to reach for it