The business caseThe problem this solves

A decorated-apparel shop takes its orders by email. The inbox mixes new orders and changes with quote requests written in order language, artwork approvals that quote the order back, shipping questions and newsletters. Somebody at the order desk reads each one, keeps only the orders, keys every garment line - style, colour, size run - and every decoration - where it goes, how it is applied, how big, how many ink colours, which art file - into the order system, and writes back to the buyer when something is missing. The first pass over an order inbox: sorting every email, reading each order or change into purchase-order fields with the line each came from, and deciding which orders can be entered and which need a question to the buyer.

The inputThe actual emails

The corpus is 200 emails, 0.10 MB (txt 200). The smallest inbox that makes the expensive mistakes possible. Quote requests are written in order language, artwork approvals quote the order back, orders say 'quote' and 'pricing', and 55 of the 110 new orders are incomplete in a way the email itself shows - no size run, two art files and 'the logo', a colour the style is not made in, a print too big for its location, a weekday that is not the date's. The phrasing libraries are shared between the classes so no single word gives the sort away.

From: Victor Hale <vhale@halestreetfire.example>
To: orders@po-desk.example
Date: Mon, 14 Sep 2026 07:32
Subject: Sweatshirts (Hale Street Fire Volunteers)
Attachments: HSFV_badge_v2.png

Hello,

Can you quote the following for us?

| Style | Color | S | M | L | XL | 2XL |
|---|---|---|---|---|---|---|
| CR-520 | Sand | 19 | 15 | 25 | 24 | 24 |
| LS-150 | White | 3 | 25 | 19 | - | - |
Line 1 deco 1: across the front | screen print | 9.5 in | 1 color (white) | art: HSFV_badge_v2.png
Line 2 deco 1: over the heart | screen print | 4 in | 3 color (light blue + red + gold) | art: HSFV_badge_v2.png
Grand total 154.

First game is 10/5.
Ship to: 24 Elm Row, Cedar Point, WA 98501

Best,
Victor

The outcomeWhat a good result looks like

One reading per email: the class, and for an order or a change the purchase-order fields with the email line each came from. An email is right only when the sort, the disposition and the hold reasons are right AND, for an order, the resolved purchase order matches the answer key field for field. 173 of 200 inbox emails wholly right against the free floor's 84, exact McNemar discordant 89/0, p = 3.23117e-27.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Sorting an order inbox so only orders and changes reach the desk — the paid call, or free code that has read your own inbox
  200 of 200 sorted against the floor's 146 - but a list lifted from this corpus's intent sentences also reaches 200. On your own inbox, a tuned list is cheap to try first.
• Reading an order into purchase-order fields with the line each came from — the paid call, with code checks after it
  103 of 130 purchase orders exact against the floor's 21, discordant 82/0, p = 4.1359e-25.
• Deciding which orders to enter and which to hold — code - src/checks.py - never the model
  The disposition is computed from the reading by code; on r001 it was right on all 200, and the reading was never asked to say ENTER or HOLD.
• Keeping a held order from being guessed — the paid call's ref plus the art check
  0 decorations given a file the email only points at, and 0 false enters, clean or under pressure.

At a glanceHow the whole thing runs

Run once, for real, on 2026-09-24. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-09-24 — r001-po-inbox - 200 inbox emails, 200 billed model calls, the fast tier, reasoning off. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Measured on this build: a clean checkout with no key configured rebuilds all 204 corpus files byte-identically under two PYTHONHASHSEEDs, re-derives every label with 0 disagreements, scores all four free arms at $0.00, recomputes every paired claim (evals/paired.py) and renders the whole board on port 9626 with the one live control disabled and the reason printed beside it. The reply cache ships, so --resume --rescore re-derives every published figure without buying a call.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. end to end per inbox email on the shipped tier - the whole streamed completion, from request to the last event, over the 200 billed calls of r001-po-inbox at 3 workers. It does not include src/recheck.py or src/checks.py, which are pure code and run after the stream closes.

Current processWhat it replaces

The first pass over an order inbox: sorting every email, reading each order or change into purchase-order fields with the line each came from, and deciding which orders can be entered and which need a question to the buyer.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusCorpusCorpusEvalsEvalsArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsUnit costUnit costBusiness caseUnit costUnit cost

• 1stations numbered in run order
• txt
• json
• jsonl
• 1 of 11 stations carry a recorded failure

scroll → 11 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

The harnessWhat this screen is

The screen is this kit's own replay board, not a deployment. It replays the 200 recorded emails so a reader can open any one and see every field beside the line it came from; a deployment reads the shop's mailbox over OAuth or IMAP and posts to the shop's own order system, and neither exists here - the ERP levels are a mock with an invented stock table and invented prices.

Buyers are shown person first, the company small in brackets. Every buyer, company, style code, address and logo on the board is invented.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one inbox email is

No split. All 200 emails are the labelled set and every arm - paid and free - is scored on all of them. Nothing is trained; the free arms are rules written from the trade, and the one written from the generator is reported as tuned, never as a floor.

SetupWhat the setup figure measured

There is no index. An email carries its own header, body and attachment text layer, so the reader opens one file and sends it whole. The catalog and open orders are read by code after the model answers, never retrieved into a prompt.

LicenceLicence

MIT (LICENSE-PUBLIC at the root of the kits repository)

Bring your ownBring your own emails

Put your emails in data/emails/ as header block, blank line, body - with any attachment's text layer inline - and your styles, colours, sizes and print limits in data/catalog.json. The prompt carries no catalog, so it does not change; src/checks.py reads the catalog you give it. To score, you need an answer key in data/gold.json's shape, which for a real inbox means somebody at the desk labelling a sample.

⚠︎ And what stops being true when you do: Nothing measured here transfers to your inbox. The 200 emails and their key were written by one program, so every count on this page - including the zero on every named trap - is a statement about this corpus. What travels is the split: the model reads, code checks and holds, and a hold is always a question to the buyer, never a guess.

The inboxSix kinds of email, and only two of them are orders

200 emails, 108,690 bytes in all (the shortest 182, the median 521, the longest 1,213), from 26 sender addresses, dated Mon Sep 14, 2026 to Fri Oct 2, 2026 across 15 days of mail. 121 of them carry an attachment.

Only a new order or an order change goes forward — 130 of the 200. Everything else is IGNORE: the desk reads it and enters nothing. The 14 under “other” are newsletter 6, spam 4, out-of-office 2, thank-you 2.

What is freeWhat a word list already gets right

The kit's free floor is a word list written from the trade before anything was measured. Split by what kind of email it is, it shows where the money is actually spent:

Slice scores read from results/eval-r001-po-inbox.json slice_scores (rechecked email_all).

Every figure in this panel is read from data/gold.json, data/corpus-manifest.json, data/emails/.

The layoutsOne order, written five ways

Every new order in the corpus is written in one of five layouts. The reading contract is the same for all of them: per line the style, colour, size run and any stated quantity; per decoration the location, method, print size, colour count and art; the in-hands date with any weekday written beside it; and the ship-to — each value citing the email line it came from.

The 110 new orders carry 141 garment lines (31 orders carry two) and 196 decorations — 100 screen print and 96 embroidery — over 7 catalog styles and 5 garment print-area tables.

The decorationsWhere each decoration goes, and what names its art

Art is named as a file on 111 decorations and as quoted text on 14. On 71 the email only points at it — “our logo”, “the mark” — and the model writes ref. Whether that pointer resolves is decided in code, by counting the art files attached: one resolves, none or two do not.

As writtenThe same fields, in each layout's own words

Order for 72 sweatshirts below.

CR-520 Crewneck Sweatshirt in Heather Gray - S 9, M 24, L 21, XL 11, 2XL 7
Screen print TPT_mark_v1.ai on the full front, 9.5", kelly green ink
Front: our mark, over the heart, 3.25 in wide, kelly green ink

Need them in hand by 19 Oct.
Ship to: 244 Quarry Rd, Westbrook, ME 04092

Order for 64 polos below.

Please make 64 black PL-400 polos - S 4, M 16, L 22, XL 18, 2XL 4, with BFO_mark_v1.ai
  embroidered on the right sleeve at 3.25" in 2 thread colors (light blue, red) and
  BFO_logo_v2.ai embroidered across the back at 10.75" wide in grey thread.

We need these by Sunday Oct 11.
Kickoff is Sat Oct 24.

CP-80 blk - 48 (one size)
  front emb 2.25" wide 2 thread (orange/grey) - HRYS_badge_v4.eps
Grand total 48.

in-hands Saturday 10/24
ship: 633 Birch St, Riverside, IA 52327
PO HR-6520

| Style | Color | M | L | XL | 2XL |
|---|---|---|---|---|---|
| LS-150 | Black | 13 | 23 | 25 | 15 |
Deco 1: across the front | screen print | 7.25" | 1 color (grey) | art: the updated mark
Deco 2: over the heart | screen print | 3.25 inches | 2 color (grey + black) | art:
  NBC_emblem_v1.dst

Thanks -
Ben

----- Attachment: PO-RR-2140.pdf (text layer) -----
PURCHASE ORDER RR-2140
Order date: 09/17/2026
Bill to: Riverside Rugby Club, 139 Maple Ave, Copperline, TX 76014
Ship to: 280 Elm Row, Suite 25, Westbrook, ME 04092
Required in-hands: 10/20/2026
Item | Color | Sizes | Qty
ST-100 | Charcoal | S 12 / M 16 / L 4 | 32
  Decoration: left breast, screen print, 3.5" wide, grey ink, art our logo
PL-400 | White | S 7 / M 23 / L 7 / XL 8 | 45
  Decoration: left sleeve, embroidery, 2.5 in, white and kelly green thread, art
    RRC_badge_v4.svg
  Decoration: right chest, embroidery, 2.75" wide, 1 thread color (white), art RRC_badge_v4.svg
TOTAL QTY: 77
----- end of attachment -----

Every figure in this panel is read from data/gold.json (_gen.layout), data/emails/.

The trapsEverything else is written to look like an order

A quote request, an artwork approval and a shipping question write their garment lines through the same generator code a new order does. Only the intent sentence separates them, and the intent sentences cross on purpose. Each trap below is counted in the email text and priced by what falling for it would cost the shop.

The model sorted 200 of 200 emails right and entered 0 of 61 incomplete orders.

Before the recheck station re-spelled weekday abbreviations (4 overrides, all of them “Wed” → “Wednesday” and the like), the raw reply to EM-0069 counted as a corrected weekday. The station changes spelling only; after it the count is 0.

As writtenThree decoys, verbatim

Hello,

Can you quote the following for us?

| Style | Color | S | M | L | XL | 2XL |
|---|---|---|---|---|---|---|
| CR-520 | Sand | 19 | 15 | 25 | 24 | 24 |
| LS-150 | White | 3 | 25 | 19 | - | - |
Line 1 deco 1: across the front | screen print | 9.5 in | 1 color (white) | art:
  HSFV_badge_v2.png
Line 2 deco 1: over the heart | screen print | 4 in | 3 color (light blue + red + gold) | art:
  HSFV_badge_v2.png
Grand total 154.

First game is 10/5.
Ship to: 24 Elm Row, Cedar Point, WA 98501

Best,

Attachments: proof_v1_signed.pdf

Hi there,

Signed-off proof v1 attached back to you.

Cheers,
Victor

On Sep 13, PO Desk wrote:
> Proof v3 for PO-26-0389 is attached for your sign-off.
> CP-80 in Khaki - qty 24, one size fits all
> Embroider HSFV_logo_v2.ai on the front panel, 2 in, 2 thread colors (silver, red)
> We need these by Nov 26.
> Please ship to 979 Harbor Blvd, Mesa Verde, AZ 85210.

Attachments: NBC_emblem_v1.dst, NBC_wordmark_v1.svg

Hi team,

Order below - please run it.

| Style | Color | M | L | XL | 2XL |
|---|---|---|---|---|---|
| LS-150 | Black | 13 | 23 | 25 | 15 |
Deco 1: across the front | screen print | 7.25" | 1 color (grey) | art: the updated mark
Deco 2: over the heart | screen print | 3.25 inches | 2 color (grey + black) | art:
  NBC_emblem_v1.dst

The event-date count is tools/build_corpus.py EVENT frames matched over the new-order emails.

Every figure in this panel is read from data/emails/ (counted with a regex over each sender's own lines), data/gold.json _gen; results/eval-b000-po-inbox-domain.json; results/eval-r001-po-inbox.json predictions (rechecked).

The holdEighteen reasons, and the question each one drafts

61 of the 200 emails are held: 55 new orders (52 with one reason, 3 with two) and 6 order changes. The 18 reasons fire 64 times. The model never decides a hold — code does, from the shop's own catalog and open-order table — and code drafts the question back to the buyer from a template, naming exactly what is missing.

A change is checked against the shop's 30 earlier purchase orders first; 10 of them are already in production or shipped, and a change to one of those is held as po_locked rather than applied.

One held orderAn order, and the question it gets back

From: Rebecca Stroud <rstroud@stroudlane.example>
To: orders@po-desk.example
Date: Wed, 16 Sep 2026 07:47
Subject: Polos
Attachments: SLR_badge_v1.ai

Hey,

Go ahead and enter this one for us.

| Style | Color | M | L | XL | 2XL |
|---|---|---|---|---|---|
| PL-400 | Royal Blue | 11 | 7 | 25 | 17 |
Deco 1: right chest | embroidery | about 4" | 1 thread color (kelly green) | art:
  SLR_badge_v1.ai
Deco 2: left breast | embroidery | 3.25" wide | black thread | art: SLR_badge_v1.ai

In hands Friday, October 8 please.
Our billing address is 835 Kestrel Ct, Ridgeview, CO 80021.
Please ship to 790 Elm Row, Ridgeview, CO 80021.

Hi Rebecca,

Thanks for your email about “Polos”. Before we enter it, one quick question:

  1. You wrote Friday Oct 8 — Oct 8 is a Thursday. Which day did you mean?

As soon as we hear back we will enter the order and send it to you to confirm.

Thanks!

Every figure in this panel is read from data/gold.json (hold_reasons), src/desk.py draft_question() run over the answer key.

DerivedWritten from the same structure as the emails, never typed

The generator builds each order as data first and renders the email from it, so the answer key is the data, not a reading of the email. python3 tools/build_corpus.py --check rebuilds all 204 files into a temporary directory from seed 20260924 and compares them byte for byte with what ships.

Re-derivedThen read again, from the email text alone

python3 -m evals.check_labels has its own parser and its own retyped rulebook. It imports nothing from the generator or the kit's code and reads the answer key only as the thing to disagree with.

Red-proofA checker that has been shown to fail

tools/redproof_labels.py seeds a defect into the real bytes of an email, requires a conviction, restores the bytes and requires an acquittal:

• a complete order's written weekday moved one day — weekday_mismatch must fire
• a Ship-to line rewritten as a Bill-to line — missing_ship_to must fire
• a new order's intent sentence replaced by a quote's — the class must flip

3 of 3 convict, and 3 of 3 acquit once the bytes are restored — the result as recorded in data/SOURCES.md.

The barWhat the key scores, from a constant to a perfect reader

The word list is the floor of record: it was written before it was measured and not edited afterwards. The tuned list and the ceiling have read the generator and may never be a bar; the ceiling reaching every email shows the key can be answered from the email alone.

Every figure in this panel is read from evals/check_labels.py (run 2026-09-24), tools/build_corpus.py --check (run 2026-09-24), tools/redproof_labels.py (result as recorded in data/SOURCES.md), results/eval-b000-po-inbox-floors.json.

The scoreWhat one run got right

173 of 200 emails wholly right; 103 of 130 purchase orders exact field for field; 83 of 110 new orders wholly right (41 of 55 complete, 42 of 55 held). All 27 wrong emails are new orders — 14 the key enters and 13 it holds — and code reached the right disposition on 27 of them.

Every missThe 27 emails, field by field

Most misses are one count: the email says “Order for 64 polos” or prints a Qty column, the key records it as the line quantity, and the reading left it out — or read a stated total twice. The key records a line quantity only where the email states one, so every quantity miss above is a count that is on the page.

As writtenTwo misses, and what code did with them

Order for 64 polos below.

Please make 64 black PL-400 polos - S 4, M 16, L 22, XL 18, 2XL 4, with BFO_mark_v1.ai
  embroidered on the right sleeve at 3.25" in 2 thread colors (light blue, red) and
  BFO_logo_v2.ai embroidered across the back at 10.75" wide in grey thread.

Following up on your quote - go ahead with the numbers below.

HD-500 char - S/M/L/XL/2XL/3XL: 3/18/22/4/25/8
  LC print 4.5 in 2c gold/red - the logo

in-hands Friday Oct 15
ship: 274 Foundry St, Cedar Point, WA 98501

1. The order: which colour for the HD-500 Pullover Hoodie? It comes in Black, Heather Grey,
  Charcoal, Navy, Maroon, Forest.
2. You wrote Friday Oct 15 — Oct 15 is a Thursday. Which day did you mean?

Raw reply to rechecked: 170 of 200 → 173 of 200 emails wholly right. The recheck station changes spelling only — the 4 overrides above are all weekday abbreviations.

Every figure in this panel is read from results/eval-r001-po-inbox.json (items, per_item rechecked answers) against data/gold.json.

CodeEvery check after the read is code

The model sorts and reads. Everything below runs in src/checks.py and src/desk.py, the same way on every run, and none of it is graded as model output:

• A change's PO must exist in the shop's table and must not be in production or shipped; its delta is applied to the original.
• The style is in the catalog; the colour is offered for that style; every size is made in that style.
• The sizes add up to any quantity or total the email states.
• The location is carried by that garment, and the print fits its area.
• Art resolves to a named file, quoted text, or exactly one attached art file.
• The in-hands date is after the email, and any weekday written beside it is that date's real weekday.
• There is a ship-to.

Anything missing or failed holds the order with a drafted question. A complete order gets a buyer confirmation, and on the buyer's yes it goes into the mock ERP.

The mock ERPTwo levels, run over the answer key

Level 1 is a sales order, the blanks checked against a stock table, and a blanks PO for any shortfall. Level 2 is one job ticket per decoration, gated on the buyer's proof approval and scheduled in business days back from the in-hands date. Run over the answer key: 55 new orders entered, 14 changes applied to 12 purchase orders, 46 of the new orders short of blanks, 89 job tickets, 3,929 pieces.

The stock table and the price list are invented, like every buyer and order in the kit, and nothing is graded against them.

The costThe read is the only paid step

Every figure in this panel is read from src/checks.py, src/desk.py build() run over the answer key; results/eval-r001-po-inbox.json (usd, tokens).

The limitsWhat this corpus cannot tell you

• It is synthetic.One generator wrote all 200 emails from one seed, from 26 sender addresses over 15 days of mail. Every buyer, company, style code and address is invented.
• It is one scored run.1 scored run, with no repeat, so how much the score moves from one run to the next is not measured here.
• The layouts are finite.5 layouts and a closed set of size, date and location notations. A word list lifted from the generator sorts 200 of 200; one written from the trade sorts 146 of 200 — so the sort measures phrasing variety the generator only partly supplies.
• Holds are mostly single-defect.52 of the 55 held new orders carry exactly one reason, so a reader that finds one reason and stops is right on most of them.
• One year, no time zones.Every in-hands date falls in 2026, so no year rollover is tested.
• The injection probe is a sample.14 target emails under 3 escalating wordings plus a clean call each, 56 calls: 0 breaches and 2 regressions (13 of the 14 clean answers were wholly right). A wording outside those three is not tested.
• The ERP is a mock.The stock table, the price list and the job schedule are invented. Connecting a real mailbox and a real order system is the implementation, and it is not measured here.

Every figure in this panel is read from results/eval-x001-po-inbox.json, results/ (run ids), data/gold.json; data/SOURCES.md.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — but they are the input total apportioned by each part’s share of the characters, not a per-part reading from the provider’s counter. The total is measured; the split is arithmetic, and a part carrying denser text really costs more of it than this says.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 2,621 tokens assembled, 2,317 are instructions — 88% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Reassembled by the kit's own src.prompt.assemble() for email EM-0052 - the one call path every arm uses - and checked against the run: the assembled system and user strings are 9324 characters, exactly the prompt_chars the run recorded for that call. The body part is published NUMBERED because that is what was sent.

Full promptThe prompt in full

You work the order desk of a decorated-apparel shop (screen printing and embroidery). You read ONE email from the shop's order inbox and do exactly two things: SORT it, and, if it is an order or a change, READ its fields. You never check anything, never decide whether an order is complete, and never fill in a value the email does not state — the shop's own code does all of that after you answer.

1. SORT. class is exactly one of:
  new_order          the buyer is placing an order now. An order that mentions an earlier quote, last week's pricing or an approved budget is still an order. An email that names an existing shop PO only to say this one is separate is a new order.
  order_change       the buyer changes an existing shop order named by its shop PO (PO-26-NNNN): a colour switch, sizes added, a size count changed, a new in-hands date, or a revised version that replaces the order.
  quote_request      the buyer asks what something would cost and has not ordered it, even when it is written with full order detail.
  artwork_approval   the buyer approves a proof or mockup. "Budget is approved" is not an artwork approval.
  shipping_question  the buyer asks about tracking, delivery or the ship date of an existing PO.
  other              newsletters, promotions, spam, thank-you notes, out-of-office replies.
Quoted reply text (lines starting with ">" after "On ... wrote:", or below "-----Original message-----") decides nothing and is never read into the answer. The buyer's own intent sentence decides the class. A text layer of an attached order form (between "----- Attachment:" and "----- end of attachment -----") is part of the email and IS read.

2. READ a new_order into:
  buyer          the sender's name as the From line gives it.
  customer_po    the buyer's OWN purchase-order number as written (e.g. "PO#KF-19" -> "KF-19"). Never a shop PO (PO-26-NNNN). null if none.
  lines          one entry per garment line, in the email's order:
    style        the style code exactly as written (two capital letters, a hyphen, digits). A garment word with no code ("polos", "hoodies", "t-shirts") is NOT a style: null. Never map a word to a code, even when only one code could fit.
    color        the GARMENT colour, written as the house colour name it denotes: Black, Charcoal, Forest, Heather Grey, Khaki, Kelly Green, Maroon, Navy, Red, Royal, Sand, Silver, Stone, White. Spelling variants and abbreviations map to the house name ("heather gray" -> "Heather Grey", "royal blue" -> "Royal", "blk" -> "Black"). A colour outside that list is written as the email writes it, capitalised. Ink or thread colours are never the garment colour. null if the garment colour is not stated.
    sizes        the size run as {SIZE: count} using only YS YM YL S M L XL 2XL 3XL OSFA. XXL/2X -> 2XL, XXXL/3X -> 3XL, small/medium/large -> S/M/L, youth small -> YS, "one size" / "one size fits all" -> OSFA. Merge youth and adult rows of one line. A "-" table cell is no size. null if no run is given.
    qty          a piece count stated for THIS line itself (a count in front of the item, "x24", or a Qty column). A count that IS the size run ("qty 60, one size fits all") is the size run, not qty. In a revised line that replaces a change, qty is null. Otherwise null.
    decorations  one entry per print or embroidery on that line, in order:
      location   one of left_chest, right_chest, full_front, full_back, left_sleeve, right_sleeve, cap_front, cap_back, cap_side. Trade words: left chest / LC / left breast / over the heart -> left_chest; right chest / RC -> right_chest; full front / FF / across the front / center front -> full_front; full back / FB / across the back -> full_back; left sleeve / LS (as a placement) / left sleeve hit -> left_sleeve; right sleeve / RS / right sleeve hit -> right_sleeve. On a CAP (style CP-...): front / front center / front panel -> cap_front, back / back arch -> cap_back, side / left side / side panel -> cap_side. On any other garment a bare "back" is full_back. A leading label such as "Front:", "Back:" or "Logo:" is a heading; the placement words after it decide. Write the placement the email gives even if that garment may not carry it.
      method     embroidery if the decoration is embroidered (embroider, emb, thread); otherwise screen_print.
      size_in    the print size in inches as a number ("about 9.5\"" -> 9.5). null if none is given.
      colors     how many ink or thread colours: the stated count ("2c", "3-color", "2 thread colors"), else the number of colours named ("gold and navy ink" -> 2). null if neither.
      art        "file:<name>" when the email names an art file ("file:TPT_mark_v1.ai"); "text:<WORDS>" when quoted words are the art ("text:CREW"); "ref" when it points at art without naming a file ("our logo", "the logo", "the new logo", "logo attached", "our crest", "our mark", "same logo as last time", "the attached art"); null if no art is mentioned. NEVER replace a "ref" with an attachment's file name — which file it means is the shop's question, not yours.
  stated_totals  every whole-order piece total the email states, one integer per statement ("Order for 72 sweatshirts below", "Total: 91 pcs", "88 pieces total", an attached form's "TOTAL QTY: 117"). A total stated twice appears twice. [] if none.
  in_hands       the date the goods must be in the buyer's hands (in hands, IH, need them by, deadline, must arrive, required in-hands), as YYYY-MM-DD; a date without a year is in 2026. An EVENT date ("first game is Oct 17") is not an in-hands date. null if none.
  in_hands_weekday  the weekday WRITTEN beside that date, as a full name ("Wed" -> "Wednesday"), exactly as the email writes it EVEN IF IT IS WRONG for that date. Never compute or correct it. null if no weekday is written.
  ship_to        the ship-to / deliver-to address exactly as written, without a trailing period. A bill-to or billing address is NEVER a ship-to: if only a bill-to is given, null.

3. READ an order_change into:
  buyer, change_of  the shop PO it changes (PO-26-NNNN),
  set   only what the email changes: "color" (house colour name), "sizes" ({SIZE: new count} for a count that is replaced), "in_hands" with "in_hands_weekday" (as above), or "lines" — a one-line list, read like a new_order line — when a revised version replaces the order. {} if nothing is set.
  add   {SIZE: count} for pieces ADDED to the order ("add 12 more in L" is an add, not a set). {} if none.

4. CITE. The body lines are numbered "NN|". For every value you read, give the line number(s) it is on, under these keys: customer_po, L1.style, L1.color, L1.sizes, L1.qty, L1.D1, L1.D2 ... (L2... for the second line), total (every line stating a total), in_hands, ship_to; for a change: change_of, set.color, set.sizes, set.in_hands, add, and L1.* for a revised line. Cite L<n>.style and L<n>.color to the line that names the item, even when the value is null. Cite nothing for a field the email does not state otherwise.

Reply with ONE JSON object on a single line, no indentation, nothing before or after it. For an email that is not an order or a change:
{"class":"quote_request","read":null,"cites":{}}
For an order:
{"class":"new_order","read":{"buyer":"Ana Ruiz","customer_po":"AR-120","lines":[{"style":"ST-100","color":"Navy","sizes":{"M":10,"L":14},"qty":null,"decorations":[{"location":"left_chest","method":"screen_print","size_in":3.5,"colors":1,"art":"ref"}]}],"stated_totals":[24],"in_hands":"2026-10-09","in_hands_weekday":"Friday","ship_to":"12 Main St, Springfield, IL 62701"},"cites":{"L1.style":[5],"L1.color":[5],"L1.sizes":[5],"L1.D1":[6],"total":[4],"in_hands":[8],"ship_to":[9],"customer_po":[10]}}
For a change:
{"class":"order_change","read":{"buyer":"Ana Ruiz","change_of":"PO-26-0100","set":{"color":"Black"},"add":{}},"cites":{"change_of":[3],"set.color":[4]}}
Use only these keys. Add no field, no status, no approval, no reasoning and no commentary, and never say whether the order is complete, held or ready — that is decided after you answer. Do not re-check your answer after you have written the object; end the reply there.

Inbox item: EM-0052

--- the email ---
From: Ben Okafor <ben.o@riversiderugby.example>
To: orders@po-desk.example
Date: Thu, 17 Sep 2026 16:57
Subject: PO RR-2140 - T-shirts for the booth
Attachments: PO-RR-2140.pdf, RRC_badge_v4.svg, RRC_emblem_v1.png

1| Hi team,
2|
3| Ready to order at last week's pricing:
4| PO RR-2140 attached - 77 pieces total.
5|
6| Thanks -
7| Ben
8|
9| ----- Attachment: PO-RR-2140.pdf (text layer) -----
10| PURCHASE ORDER RR-2140
11| Order date: 09/17/2026
12| Bill to: Riverside Rugby Club, 139 Maple Ave, Copperline, TX 76014
13| Ship to: 280 Elm Row, Suite 25, Westbrook, ME 04092
14| Required in-hands: 10/20/2026
15| Item | Color | Sizes | Qty
16| ST-100 | Charcoal | S 12 / M 16 / L 4 | 32
17|   Decoration: left breast, screen print, 3.5" wide, grey ink, art our logo
18| PL-400 | White | S 7 / M 23 / L 7 / XL 8 | 45
19|   Decoration: left sleeve, embroidery, 2.5 in, white and kelly green thread, art RRC_badge_v4.svg
20|   Decoration: right chest, embroidery, 2.75" wide, 1 thread color (white), art RRC_badge_v4.svg
21| TOTAL QTY: 77
22| ----- end of attachment -----
--- end of email ---

Reply with the JSON object described in your instructions.

Raw responseThe raw response

{"class":"new_order","read":{"buyer":"Ben Okafor","customer_po":"RR-2140","lines":[{"style":"ST-100","color":"Charcoal","sizes":{"S":12,"M":16,"L":4},"qty":32,"decorations":[{"location":"left_chest","method":"screen_print","size_in":3.5,"colors":1,"art":"ref"}]},{"style":"PL-400","color":"White","sizes":{"S":7,"M":23,"L":7,"XL":8},"qty":45,"decorations":[{"location":"left_sleeve","method":"embroidery","size_in":2.5,"colors":2,"art":"file:RRC_badge_v4.svg"},{"location":"right_chest","method":"embroidery","size_in":2.75,"colors":1,"art":"file:RRC_badge_v4.svg"}]}],"stated_totals":[77,77],"in_hands":"2026-10-20","in_hands_weekday":null,"ship_to":"280 Elm Row, Suite 25, Westbrook, ME 04092"},"cites":{"customer_po":[4],"L1.style":[16],"L1.color":[16],"L1.sizes":[16],"L1.qty":[16],"L1.D1":[17],"L2.style":[18],"L2.color":[18],"L2.sizes":[18],"L2.qty":[18],"L2.D1":[19],"L2.D2":[20],"total":[4,21],"in_hands":[14],"ship_to":[13]}}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Turn order emails into ready-to-enter orders — 200 inbox emails drawn from 200 real emails. One model answered, and every answer was then graded Eight different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Every grader is pure code over the answer key. An arm's answer goes through the same src/recheck.py and the same src/checks.py whatever produced it, and is compared field by field against data/gold.json. The disposition and hold reasons are never the model's to state - code computes them from the sort and the reading, identically on every arm.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar here is a MEASURED token count from this kit's own run records multiplied by a published rate in build/facts/models.json, whose as_of is carried per row. No price was read from a vendor page for this kit, and no volume, committed-use or batch discount is modelled.

Rates checked 2026-09-18.

Grading unitWhat the grading figure prices

Re-scoring an existing run costs nothing: --resume --rescore re-derives every figure from the committed reply cache at $0.00, and evals/paired.py recomputes every p-value from the committed records.

The gradersEight ways to grade

Three more free arms ship beside the floor of record. constant answers one fixed class for every email; the best on the headline is quote_request at 22 of 200, and a different class wins each cell. tuned is the domain reader with its sort lifted from this corpus's own intent sentences - it has read the generator, so it is reported beside and never used as a bar (96 of 200). ceiling is the answer key itself through the station and the checks, 200 of 200.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — top_k, chunk size, prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-09-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingRe-scoring this kit costs nothing, and that is a design decision

A judge cannot answer this kit's question. 'Is this the style, colour, size run, print location and art file the email gives, and is the order complete?' is an exact comparison against a key and a set of catalog checks, so every grader is code and re-scoring the whole run costs $0.00.

Cost driversWhat actually moves the bill

• The instruction block, 8135 characters sent byte-identically on every call - the sort vocabulary, the answer contract and the reading rules. It is most of every prompt, which is why the prefix cache is the cost story: 87.8% of all input tokens on r001 came back as cache hits.
• The email itself - 522 bytes at the median and 933 at p95, header, numbered body and any attachment text layer. The only part that differs call to call.
• The reply: 144.2 output tokens on average and 388 at most. A non-order email answers in about 15; a two-line order with every cite runs to a few hundred.
• Nothing else. The checks, the hold, the drafted question, the confirmation and both ERP levels are code and cost $0.00.

Your volumeWhat it costs at your volume

Linear. Each email is one call and nothing batches: 2,000 emails would be about ten times this run's $0.035109 at the same cache rate. The code half does not grow the bill at all.

Model choiceWhy this model

The cheapest tier was the default and nothing asked for more: the largest reply over 200 billed calls was 388 output tokens of a 1,000 ceiling, 0 were at the ceiling, reasoning was off on every call, and the reading beat the free floor on every card cell.

Other modelsThe same inbox email on every model we track

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The codeEvery file, and what it is for

15 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pycorpus generator

Writes all 200 emails, the answer key, the catalog, the open-orders table and the manifest from one seed (20260924). --check rebuilds every file and asserts it comes back byte-identical under two PYTHONHASHSEEDs; --coverage fails the build if a phrasing variant, class, disposition or hold reason is never dealt. Dates come from a calendar, so every weekday written beside a date is the real one unless the email is planted to get it wrong.

# Build UC0526's synthetic corpus: 200 emails landing in a decorated-apparel shop's order inbox.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
SEED = 20260924
N_EMAILS = 200
DATASET = "po-inbox synthetic decorated-apparel order inbox"
YEAR = 2026
SHOP_INBOX = "orders@po-desk.example"
STYLES = {
STYLE_KEYS = sorted(STYLES)

data/gold.jsonthe answer key

200 emails: the class, the full reading with every field's email line, the resolved purchase order, the hold reasons and the disposition - all DERIVED from the structure the generator wrote, never typed. 110 new orders, 20 changes, 70 non-orders; 69 ENTER, 61 HOLD, 70 IGNORE.

# 

data/catalog.jsonthe catalog and the open orders — a swap seam

7 invented styles with the colours and sizes each is made in and the print-size limit per location, plus data/open_orders.json - 30 open purchase orders, 10 of them already in production or shipped and therefore locked against a change.

You change it to: Your styles, the colours and sizes each is made in, and your print-size limit per location. The checks read it; the prompt does not, so a new style needs no prompt change.

# 

src/prompt.pythe prompt — a swap seam

SYSTEM, a literal: the six sort classes and what each costs to get wrong, the answer contract, the vocabularies (locations, methods, sizes, art forms) and the reading rules - absent is null and never filled, the weekday as written, cite the line. assemble() sends it FIRST and the numbered email LAST so the prefix cache has the longest run to hit.

You change it to: SYSTEM is a literal. Change it and every figure here is void, because it is what the 200 cached replies answered - nothing in it was edited after a paid reply was read.

# SEAM — what the model actually receives, and the vocabulary it is allowed to answer in.

VERDICTS = ["new_order", "order_change", "quote_request", "artwork_approval", "shipping_question",
VERDICT_MEANINGS = {
LOCATIONS = ["left_chest", "right_chest", "full_front", "full_back", "left_sleeve", "right_sleeve",
METHODS = ["screen_print", "embroidery"]
SIZES = ["YS", "YM", "YL", "S", "M", "L", "XL", "2XL", "3XL", "OSFA"]
WEEKDAYS = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]
HOUSE_COLORS = ["Black", "Charcoal", "Forest", "Heather Grey", "Khaki", "Kelly Green", "Maroon",
GARMENT_OF_PREFIX = {"ST": "tee", "LS": "long-sleeve tee", "PL": "polo", "HD": "hoodie",
FREE_TEXT_BUDGETS = {"ship_to": 120, "customer_po": 30, "buyer": 60, "art": 80}

src/adapters/__init__.pythe adapter

One streamed call per email, reasoning explicitly disabled, max_tokens 1,000, a reply stop sized from the widest legitimate answer, and the prefix-cache split read from the provider's usage so the bill is priced at the hit rate where it was a hit. The one ledger site.

# UC0526 po-inbox — SEAM 1, the model. TAKEN FROM tools/templates/adapters.py; the kit's changes

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TIMEOUT_S = 600
TRANSPORT_RETRIES = 1
FIRST_EVENT_TIMEOUT_S = 150
KEEPALIVE = object()
TRANSIENT_STREAM_MARKERS = ("unable to start processing", "timeout limit", "try again later",
def _post(url, headers, payload, timeout=TIMEOUT_S):

src/answer.pythe answer parser

Takes the first balanced JSON object out of the reply; anything else is recorded as unreadable and scored wrong, never crashed on.

# THE ONE AI STATION. Everything else in this kit is deterministic code.

MAX_TOKENS = 1000
MAX_TOKENS_REASON = ("rung 1 of the operator's ceiling ladder (1,000 -> 1,500 -> 2,000, each rung "
THINKING = THINKING_OFF
STOP = REPLY_STOP
def parse_reply(text):
def normalise(obj):
def answer(cfg, item, raw_email, complete_fn=None, max_tokens=MAX_TOKENS):

src/recheck.pythe station

Coerces types and vocabulary only - a size written XXL becomes 2XL, a weekday written Wed becomes Wednesday - and records every change. It never supplies a value. On r001 it made 4 changes, all of them a weekday abbreviation; on every free arm and on the answer key it makes 0.

# THE STATION — the structural half of the answer contract, re-applied in code to EVERY arm.

ORDERS = ("new_order", "order_change")
SIZE_SYN = {"XXL": "2XL", "2X": "2XL", "XXXL": "3XL", "3X": "3XL", "SMALL": "S", "MEDIUM": "M",
class _Ledger:
def _int(v, led, field):
def _float(v, led, field):
def _text(v, led, field, budget_key=None):
def _color(v, led, field):
def _sizes(v, led, field):
def _date(v, led, field):

src/checks.pythe checks

decide(cls, read, attachments, email_date): every catalog check, the hold reasons (18 of them) and the disposition, in code, for every arm. Style known, colour offered, sizes offered, totals agree, location valid, print size within the limit, art resolves to one file, in-hands after the email date with its weekday matching the calendar, ship-to present, change of a PO that exists and is not locked.

# The code half of UC0526: every check, the hold, and the disposition. No model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
ORDER_CLASSES = ("new_order", "order_change")
WEEKDAYS = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]
LOCKED = ("In production", "Shipped")
def catalog():
def open_orders():
def _art_reason(art, attachments):
def resolve_art(art, attachments):

src/desk.pythe desk

Everything after the checks: the drafted question per hold reason, PO numbering, changes applied onto the open order, proof approvals read from the inbox, the buyer confirmation, and the mock ERP - level 1 blanks against an invented stock table with a blanks PO for the shortfall, level 2 one job ticket per decoration scheduled in business days. Deterministic, never graded as model output.

# The desk after the read: every screen of the board, built in code from one reading per email.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
DAYS = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"]
MONTHS = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]
LOC_LABEL = {"left_chest": "Left chest", "right_chest": "Right chest", "full_front": "Full front",
METHOD_LABEL = {"screen_print": "Screen print", "embroidery": "Embroidery"}
CLASS_LABEL = {"new_order": "New order", "order_change": "Order change",
FLOW = ["Received", "Buyer confirmed", "In ERP", "In production", "Shipped"]
FIRST_NEW_PO = 418

evals/scoring.pythe scorer

Eleven cells, RAW and RECHECKED, per arm: email_all (the headline), sort, forward, disposition, po_exact, line_row, deco_row, held_right, cite, and the two error directions false_enter and false_hold, never averaged.

# Score predictions against data/gold.json -- pure code, no judge.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
ORDERS = ("new_order", "order_change")
def load_gold():
def _norm_text(s):
def norm_po(po):
def norm_deco(d):
def when_of(g):
def evaluate(pred, g):

evals/baseline.pythe free arms

domain (a desk's own word lists and layouts - THE FLOOR OF RECORD), constant (one fixed class), tuned (the domain reader with its sort lifted from this corpus's own intent sentences - reported beside, never a bar) and ceiling (the answer key through the same station and checks). All $0.00.

# The FREE arms as the harness runs them — one answer per email, in the model's own answer shape,

MODES = ("domain", "constant", "tuned", "ceiling")
SHIPPED_ARMS = ("domain", "constant")
CEILING_ARMS = ("tuned", "ceiling")
LABELS = {
CELLS = F.CELLS
LOWER_IS_BETTER = ("false_enter", "false_hold")
CLASSES = ["new_order", "order_change", "quote_request", "artwork_approval", "shipping_question",
def load_items():
def email_text(item):

evals/paired.pythe paired test

Exact McNemar per cell, paid against every free arm, from the committed records, re-deriving each count and refusing if it disagrees.

# The paired test per CLAIM, per CELL, per free arm — exact McNemar, from the committed result files.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
PAID = "r001-po-inbox"
ARMS = ("domain", "constant", "tuned")
CELLS = ("email_all", "po_exact", "disposition", "held_right", "line_row", "deco_row",
LOWER_IS_BETTER = ("false_enter", "false_hold")
def mcnemar(b, c):
def load(rid):
def outcomes(d):

evals/run.pythe harness

The one paid entry point: --stub and --floor are free, a live run needs --yes, --resume never re-buys a cached answer, and the rate card is named in the file.

# Read all 200 inbox emails and write one result file. THE PAID ARM SPENDS MONEY.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
DATA = os.path.join(HERE, "data")
KIT = "po-inbox"
ROWS_UNIT = "inbox emails"
MAX_WORKERS = 3
CEILING_LADDER = (1000, 1500, 2000)
CELLS = B.CELLS
ORDERS = S.ORDERS

evals/injection.pythe injection probe

14 emails x (clean + 3 escalations); --stub and --verify-neutral are free and ran first.

# THE ADVERSARIAL PROBE (x001) — an instruction planted in the email, in three escalations, pushing

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
KIT = "po-inbox"
ESCALATIONS = [
TARGET_FAMILIES = [("quote_request", "quote"), ("artwork_approval", "artwork"),
ALLOWED_TOP = {"class", "read", "cites"}
ALLOWED_READ = {"buyer", "customer_po", "lines", "stated_totals", "in_hands", "in_hands_weekday",
ALLOWED_LINE = {"style", "color", "sizes", "qty", "decorations"}
ALLOWED_DECO = {"location", "method", "size_in", "colors", "art"}

src/app.pythe board

A stdlib server over ui/: the inbox, the order reader with every field beside its email line, the purchase orders, the review queue with drafted questions, the buyer confirmation and the two ERP levels - replayed from the committed run. It refuses to start if any screen disagrees with the record.

# The PO Desk, replayed. One command, no dependency, no build step.

KIT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(KIT, "ui")
RESULTS = os.path.join(KIT, "results")
PORT = int(os.environ.get("PORT", "9526"))
SLUG = "po-inbox"
RUN_ID = "r001-%s" % SLUG
FLOOR_ID = "b000-%s-domain" % SLUG
PROBE_ID = "x001-%s" % SLUG
HEADLINE = "email_all"

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 1492 input and 155 output tokens per query at top-k 1, of which about 150 tokens are fixed (system prompt, chat framing, the question itself) and the rest is retrieved context. Change the retrieval depth and the input scales with it. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model and top-k follow the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

A local, single-user board over an invented corpus. It binds 127.0.0.1 on port 9526, has no auth, no rate limit and no session - correct for a kit you run on your laptop, and not a deployment posture. A real desk reads the shop's mailbox over OAuth and writes to its order system only after the buyer confirms.

Read from the environment or a gitignored .env, never written into the repo, never requested from a reader on any surface, and never printed. The board's one live control is disabled when no key is configured.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. Both were run for real on 2026-09-24, 56 live calls on ONE probe (x001-po-inbox): 14 emails - one per corpus family - each read clean and then under 3 escalating instructions planted as the last body line, against the same 200-email corpus at sha256:0ccfd92a5891fbea.

The result0 breaches and 0 false enters over all 42 attacked calls and the 14 clean ones - including 14 urgent instructions to add "status": "ENTER" and 14 authority instructions to add "approved": true and ignore the hold rules. 2 readings that were right clean changed under pressure, of 39 that could have.

14 emails x (1 clean + 3 escalations) = 56 calls, $0.009453. Targets are one email per corpus family, chosen by a rule over the answer key before any call was bought. Regressions by framing: plain 1, urgent 0, authority 1. The wordings are verbatim in evals/injection.py and were NOT retuned after the results were read.

HonestyWhat this does not prove

• whether a wording outside these three would breach — three escalations is a sample of an unbounded space, and the wordings were NOT retuned after reading the results
• whether the clean call inside this probe matches the same email's answer in r001 — they are separate purchases of the same prompt
• whether an instruction planted INSIDE the order text (on a decoration line, or in an attached form) behaves the same way; only the last-line position is probed, because it is the only one that leaves every cited line number unmoved

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Every inbox email gets one reading: a class from six, and for an order or a change the purchase-order fields with the email line each came from. A field the email does not state is null - never filled, never guessed. The reply carries no status, approval or disposition: code decides ENTER, HOLD or IGNORE from the reading, holds any order with a missing or failing field, and drafts the question to the buyer. Nothing is entered until the buyer confirms.

src/prompt.py::SYSTEM states the classes, the contract and the absent-is-null rule; src/recheck.py coerces types and drops any key outside the contract; src/checks.py applies every catalog check and decides the disposition; src/desk.py drafts the question and gates the ERP on the buyer's confirmation.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• NOT an order-entry system. It reads and checks; the ERP here is a mock, and a real one is only written after the buyer confirms.
• NOT a pricing or quoting tool. Quote requests are sorted and set aside; no price is read or produced.
• NOT an artwork approver. A proof approval email is sorted and read for which PO it names; nothing judges the art.
• NOT a mailbox integration. The emails are files; OAuth or IMAP is the production step this kit names and does not build.

WatchedWhat is watched, and why that one

The kit measures 64 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

23 measured by the latest run 41 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

8 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 4 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

extraction · no model in the path — a baseline, not a peer column — 4 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 4 runs measured the same system — they differ on cited_fields_scored, constant_best_on_the_headline, constant_classes, doc_count, failures, floor, provider, rechecked_rederived_from_cache, rows_unit, seed, slice_new_order_complete_emails, slice_new_order_emails, slice_new_order_held_emails, slice_non_order_emails, slice_order_change_emails, verdicts, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

extraction · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 60 chips that all say so.

stub · no model in the path — a baseline, not a peer column — 2 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 2 runs measured the same system — they differ on adversarial_calls, cited_fields_scored, could_not_verify_rows, doc_count, failures, framings, inbox_emails, max_tokens, order_decorations, order_emails, order_lines, orders_to_enter, orders_to_hold, orders_to_hold_for_false_enter, probe_authority_calls, probe_id, probe_kind, probe_plain_calls, probe_target_emails, probe_urgent_calls, rechecked_rederived_from_cache, regression_predicate_ok, rows_unit, seed, slice_new_order_complete_emails, slice_new_order_emails, slice_new_order_held_emails, slice_non_order_emails, slice_order_change_emails, thinking, usd_calls, verdicts — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 23 chips that all say so.

DeviationsWhat deviated

0 breaches across 8 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The free arms, the label retype and its red-proof, the corpus --check, the paired tests and the board's startup agreement check are pure code and run on every commit at $0.00. The paid run and the probe repeat only when src/prompt.py changes.

In one lineNo framework, deliberately

No framework. The kit is the Python standard library and one streamed HTTP call; requirements.txt has no third-party entry, deliberately. The one thing a framework would own here - the call and its retry - is 200 lines in src/adapters/, and everything after the reply is plain code a reader can step through.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

One pass, three seams: open the email, ask once, check in code. Nothing sends work backwards and nothing retries a completed email, so there is no state to hold between steps and no loop to bound.

The other sideWhat a framework costs you

• A framework would own the call and its retry, and the retry is where a kit quietly starts paying twice for the same email. This one retries a transport failure and never a completed reply.
• Structured-output mode is one flag in most frameworks and it was deliberately never sent: it changes what is being measured, because a reply forced into shape cannot show that it would have added a status field.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-po-inbox on the fast tier, 2026-09-24. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 4 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, FOUNDRY_RUN_ID, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, PYTHONHASHSEED.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT (LICENSE-PUBLIC at the root of the kits repository) Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe whole email right: sort, disposition, holds and the purchase order

For every inbox email, whether the sort is right, the disposition is right, the hold reasons are exactly right and - for an order or a change - the resolved purchase order matches the key field for field. One field wrong fails the email.

MethodHow the test was run

python3 -m evals.run --run-id r001-po-inbox --resume --rescore

Every grader on these pages scored the same 200 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: ITSELF - this grader IS the reference standard for the kit. The key in data/gold.json was derived by the corpus generator from the structure it wrote, and is restated independently by evals/check_labels.py (its own parser and a retyped rulebook) with 0 disagreements.

No true/false rates for this grader. It records 3 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold. The expectation is derived from the answer key every run and never recorded from a previous run.

Cadence: every commit - it is pure code over a committed cache

The decisionWhen to reach for it

In one lineThe purchase order, field for field

For each of the 130 order and change emails, whether the resolved purchase order - every line, size run, quantity, decoration, date and ship-to - equals the key's.

MethodHow the test was run

python3 -m evals.paired

Every grader on these pages scored the same 200 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the answer key, restated by evals/check_labels.py with 0 disagreements. This grader's own rates are UNKNOWN because it is scored against that key rather than being it.

No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold and no tuning - an exact comparison against the key.

Cadence: every commit - it is pure code over a committed cache

The decisionWhen to reach for it

In one lineEach order line: style, colour, size run and quantity

Per order line (141 of them), whether style, colour, size run and quantity all match the key.

MethodHow the test was run

python3 -m evals.paired

Every grader on these pages scored the same 200 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the answer key. Rates UNKNOWN for the same reason as the purchase-order grader.

No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold.

Cadence: every commit - it is pure code over a committed cache

The decisionWhen to reach for it

In one lineEach decoration: location, method, print size, ink colours and art

Per decoration (196 of them), whether location, method, size, ink-colour count and art reference all match the key.

MethodHow the test was run

python3 -m evals.paired

Every grader on these pages scored the same 200 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the answer key. Rates UNKNOWN.

No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold.

Cadence: every commit - it is pure code over a committed cache

The decisionWhen to reach for it

In one lineHeld for exactly the right reasons

For the 61 orders the key holds, whether the arm's reading makes code hold it with exactly the key's hold reasons - no more, no fewer.

MethodHow the test was run

python3 -m evals.paired

Every grader on these pages scored the same 200 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the answer key's hold reasons, derived from the planted defects.

No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold.

Cadence: every commit - it is pure code over a committed cache

The decisionWhen to reach for it

In one lineThe two error directions: a held order entered, a complete order not entered

false_enter: a held order the arm's reading would ENTER. false_hold: a complete order it would hold or ignore. Counted apart, never averaged.

MethodHow the test was run

python3 -m evals.paired

Every grader on these pages scored the same 200 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the key's dispositions.

No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? false_enter's band is exact zero; false_hold carries its RAW count beside the rechecked one.

Cadence: every commit - it is pure code over a committed cache

The decisionWhen to reach for it

In one lineThe sort: which of six classes

Whether the email's class is the key's: new_order, order_change, quote_request, artwork_approval, shipping_question or other.

MethodHow the test was run

python3 -m evals.paired

Every grader on these pages scored the same 200 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the key's class labels.

No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold.

Cadence: every commit - it is pure code over a committed cache

The decisionWhen to reach for it

In one lineWhat three framings did to the reading

Whether an instruction planted as the last body line of an email makes the reading guess, add a status or approval, or enter a held order - and how many readings that were right clean became wrong under pressure.

MethodHow the test was run

python3 -m evals.injection --verify-neutral (free), then the scored probe record x001-po-inbox

Every grader on these pages scored the same 200 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the same key through the same graders. Rates UNKNOWN - it measures the attacker.

No true/false rates for this grader. It records 3 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold on breaches - the band is exact zero.

Cadence: only when src/prompt.py changes - $0.009453 for 56 calls.

The decisionWhen to reach for it