The business caseThe problem this solves

A buyer issues a purchase order and the supplier answers it -- or does not. The answer changes a quantity, a price or a date, or it never arrives, and either way the first place anybody notices is weeks later at the goods-in door or on an invoice that will not match. The systems that check this match the acknowledgments they RECEIVED against the orders they name, which is one join and is very good at what it does. It has one hole and the hole is the shape of the problem: a line nobody answered produces no document to match, so it is not clean and not dirty, it is absent -- and 38 of the 308 line rows in this corpus are in exactly that state. Replaces reading an acknowledgment against a purchase order line by line, and replaces the document-matching check that compares the acknowledgments that arrived and cannot see the lines that never did.

The inputThe actual PO acknowledgments

The corpus is 50 PO acknowledgments, 0.11 MB (txt 50). A real purchase order and its acknowledgment are not something this repo can publish. Together they name a manufacturer's supplier base, its negotiated unit prices item by item, its production schedule and a named buyer with a direct extension. There is no public corpus of (purchase order, acknowledgment, correct follow-up list) pairs, for the same reason there is no public corpus of bank statements. Generating it also makes the labels mechanical: gold's status, price gap and slip are all src/rulebook.py run over the same values the record itself states, never somebody's reading of the buyer's note. ⚑ AND IT IS THE ONLY WAY THE HEADLINE GRADER EXISTS AT ALL. The question this use case was filed under is whether an acknowledgment-stage mismatch predicts a DOWNSTREAM exception, which means the corpus has to carry an outcome that happens weeks after the document it ships. downstream_exception is planted per bucket in exact counts, appears in no corpus file, and is deliberately not a function of the status -- 12 lines the rulebook CLEARS fail downstream and 30 lines it FLAGS come out fine. Both generator and gold gate assert those two populations have members before any run may spend.

Purchase Order
--------------
PO-7959-DP

Order Placed On
---------------
2026-05-05

Supplier Reference
------------------
SUP-5448 -- Marchgate Metals

Acknowledgment Rulebook
-----------------------
PAR-1 -- illustrative, shipped with this kit; reproduces no real procurement rule

Order Lines
-----------
quantities are in the ordering unit of measure; prices are the trading currency's major unit
L010    MAT-39765     Hydraulic hose assembly, 1/2 inch          qty 37       at 50.31      wanted 2026-06-17
L020    MAT-61763     Oil seal 25 x 40 x 7, nitrile              qty 300      at 1.58       wanted 2026-07-06
L030    MAT-79070     Cast iron pulley, 100mm                    qty 55       at 10.02      wanted 2026-07-31
L040    MAT-17752     Pneumatic fitting, push-in, 8mm            qty 360      at 1.96       wanted 2026-07-05
L050    MAT-22477     Proximity sensor, inductive, M12           qty 65       at 34.46      wanted 2026-07-07
L060    MAT-47196     Contactor, 3 pole, 25A                     qty 50       at 71.82      wanted 2026-07-15
L070    MAT-22894     Tap, M8 x 1.25, spiral flute               qty 170      at 11.60      wanted 2026-07-23

Supplier Acknowledgment
-----------------------
L040    item MAT-17752     ack qty 260             at 1.96                 promised 2026-07-05
L060    item MAT-47196     ack qty 50              at 78.57                promised 2026-07-15
L050    item MAT-22477     ack qty 65              at 34.46                promised 2026-07-17
L020    item MAT-61763     ack qty 300             at 1.58                 promised 2026-07-12
L030    item MAT-79070     ack qty 55              at 10.02                promised -- not stated
L090    item MAT-36613     ack qty 165             at 11.19                promised 2026-08-01

The outcomeWhat a good result looks like

One follow-up list per purchase order: which lines were never answered, which came back short, mispriced or late, by how much, which are covered by a contracted price, which the record cannot answer for -- and whether the order as a whole needs somebody to DECIDE or somebody to CHASE.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Reconciling a supplier acknowledgment before the order reaches receiving, as a follow-up list for a procurement desk to work through — the one tier measured here -- 308 of 308 lines called correctly on this corpus, 0 false alarms on 148 flagged lines, and all 38 lines the supplier never answered caught
  The free incoming-document matcher gets 61.69 pct of the calls right and cannot see an unanswered line at all; it raises 62 false alarms on the same 50 records and clears 18 unreadable rows as confirmed.
• Answering the audit question the use case was filed under -- show me every PO the supplier never acknowledged, how long it sat, and what happened at receipt — the measured run for the first two, and NOTHING HERE for the third
  38 of 38 unacknowledged lines caught and named, against the free matcher's 0, and the age in days is computed and printed on every order. The third part of the question is answered by grader 5 at 84.29 pct recall -- on a PLANTED outcome, which is a measurement of the prediction problem's shape and not of any real estate's exception history.
• Running the check with no budget at all — evals/baseline.py -- the incoming-document matcher, free, no key, whole corpus in under a second
  It reads every field it emits exactly and derives both numbers correctly: 100.0 pct on the price gap and 100.0 pct on the slip, identical to the paid run on the second and better on the first. Over 270 of the 308 line rows it is a faithful reader.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-23. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-23 — r001-po-ack. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with NO key configured reproduces everything on this page except a new call: the 50 order records, the 50 gold rows, all five committed run records, and the local UI, which renders and returns a plain sentence saying nothing was called. python3 tools/build_corpus.py rebuilds the corpus and gold byte-identically from the seed; python -m evals.check_labels re-validates gold, the selector and the free floor; and python -m evals.run --run-id b000-po-ack-ackmatcher --baseline re-runs the whole floor over all 50 records in well under a second, for $0.00, and reproduces its published numbers exactly. Only --run-id r001-po-ack needs a key.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per order record, measured in evals/run.py around the whole extract call -- segment, select, prompt, the one model call, parse, span and the pure-code rule. There is no retrieval step to exclude, so this figure and the model-only figure in Cost are the same measurement; both are stated so a reader is not left inferring one from the other. Measured at 12 concurrent workers, so it is per-record latency and not throughput: the whole 50-record run took 140.6 seconds of wall clock.

Current processWhat it replaces

Replaces reading an acknowledgment against a purchase order line by line, and replaces the document-matching check that compares the acknowledgments that arrived and cannot see the lines that never did.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one order line is

no split -- one whole order record goes into one call. src/segment.py cuts each record into addressable sections on its underlined headings so a value can name WHICH side of the comparison it was read from, but nothing is chunked and nothing is dropped for length.

SetupWhat the setup figure measured

There is no index and nothing is built. Each order record goes whole into one call, so the build cost is genuinely zero rather than unmeasured -- the only preparation is tools/build_corpus.py regenerating the corpus and gold from the seed, which takes under a second and makes no call.

LicenceLicence

MIT (this repository's own licence). Nothing was fetched and nothing is licensed from anybody: no real manufacturer, supplier, part, brand, person or currency is named anywhere in the kit.

Bring your ownBring your own PO acknowledgments

Replace data/corpus/*.txt with your own orders and data/gold.jsonl with your own labels; rewrite data/rulebook.json with the terms you are actually bound by AND SUPPLY A REAL ACKNOWLEDGMENT WINDOW, which this kit does not have; adjust the section names in src/select.py to match your headings and put anything you will not send into NEVER_SENT. src/segment.py falls back to one whole-document segment when it finds no underlined headings, so a differently-shaped extract still runs -- more expensively, and correctly. Re-run and re-publish your own numbers.

⚠︎ And what stops being true when you do: Nothing measured here transfers. Three things in particular: this corpus puts BOTH SIDES of every comparison in ONE FILE under ONE line-reference scheme, so the run never has to join an acknowledgment to an order across two systems whose keys differ -- which is most of the real work; the shipped rulebook is an invention, so a perfect score means the pipeline agreed with a file this kit wrote; and downstream_exception is a planted label, so grader 5 measures the SHAPE of the prediction problem and not any real estate's exception rate.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 5,549 tokens assembled, 2,586 are contexts — 47% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Replayed from the kit's own src/prompt.py::build() rather than logged by the run, which is only trustworthy because it was checked: the replayed part sizes and the section list are identical to what the recorded call in results/example-PO-0043.json carried. The TOKEN counts beside each part are not replayed at all -- they are the provider's own prompt_tokens measured in nested prefixes by evals/prompt_tokens.py (run p001-po-ack, 4 calls at max_tokens=1), so each part's size is the difference between two counts the provider returned.

Full promptThe prompt in full

You read ONE PURCHASE ORDER AND THE SUPPLIER ACKNOWLEDGMENT RECEIVED AGAINST IT -- what a buyer asked for, line by line, and what the supplier answered -- and you extract structured fields from it. You return JSON and nothing else.

THE THREE FACTS COMPARED ARE QUANTITY, UNIT PRICE AND DELIVERY DATE. The question is whether the supplier's answer matches the order, and where it does not, whether the difference is one that is supposed to be there. A supplier who answered at all is NOT a finding -- most lines here were answered, that is what an acknowledgment is. A supplier who answered DIFFERENTLY, or who answered NOTHING, is.

You are PROPOSING a follow-up list for a qualified person to act on. You never accept or approve an acknowledgment, never write to or amend a purchase order, never contact a supplier, never send a chase, and never decide whether an acknowledged price or quantity should be accepted. The rulebook below is an ILLUSTRATIVE rulebook shipped with this kit; it is not a law, a regulation, an EDI guideline or anybody's procurement policy, and it is not an authority.

RULES, in order of importance:
1. THE ROSTER IS `Order Lines`, NOT THE ACKNOWLEDGMENT. Return one object per row of `Order Lines`, in the order that section lists them, and no others -- INCLUDING the rows the supplier never answered. A line with no acknowledgment row is not a line to leave out; it is the most important row on the page. THE `Supplier Acknowledgment` SECTION CARRIES ROWS FOR LINE REFERENCES THIS ORDER DOES NOT HAVE -- a supplier answering an older order, or a line that was cancelled. Reporting one of those is as wrong as missing a real one. Never invent a line.
2. THE TWO SECTIONS ARE IN TWO DIFFERENT ORDERS AND NEITHER MATCHES THE OTHER. Join by line reference, matched exactly and in full. L040 and L400 are different lines.
3. If the acknowledgment carries NO ROW at all for a line, every `ack_` field is null and both derived numbers are null. That is not a gap in your answer, it is a fact about the extract, and it is what makes the line unacknowledged. Never carry a value across from the order and never infer one.
4. If a row is there and a value on it is not stated, return null for that value. Do not infer it and do not use what you know about the world.
5. COPY THE ACKNOWLEDGMENT'S OWN ITEM REFERENCE VERBATIM into `ack_item_ref`. It is NOT always the same reference the order line names. Do not correct it, do not normalise it and do not copy the order's reference across -- that difference is evidence and you would be destroying it.
6. `price_gap_minor` is DERIVED AND IS NEVER STATED. It is the ABSOLUTE difference, IN MINOR UNITS (hundredths), between the ACKNOWLEDGED unit price and the ORDERED unit price -- against the ORDER, not against any contracted price. 1.28 against 1.35 is 7. Return 0 where they are equal and null where either is not recorded.
7. `slip_days` is DERIVED AND IS NEVER STATED. It is the promised date MINUS the requested date, in whole days, SIGNED. A promise EARLIER than requested is NEGATIVE, not positive, and it is not a finding. Return 0 where they match and null where either is not recorded.
8. `reconciliation` is decided ONLY by the RULEBOOK given below. Work through these seven gates IN ORDER and STOP at the first one that fires:
   a. UNACKNOWLEDGED. The acknowledgment section carries no row for this line reference at all. Answer 'unacknowledged'. THIS GATE IS FIRST because there is nothing to compare, and because silence is the answer rather than a missing input.
   b. NOT DETERMINABLE. A row exists and the acknowledged quantity, unit price or promised date is not recorded on it -- or the order line itself does not state one of its own three values. Answer 'not_determinable'. It is a real answer, not a failure to produce one; a confident wrong 'fine' is the expensive mistake here.
   c. AUTHORISED VARIANCE. A contracted unit price is on file for THIS ORDER LINE'S item, the acknowledged unit price equals it exactly, the acknowledged quantity equals the ordered quantity, AND the promised date is not more than 2 day(s) later than requested. Answer 'authorised_variance'. THIS GATE IS THIRD. An order raised at a stale price against a contract that has since moved comes back at the contracted price, correctly, and calling that a mismatch is the error that gets a follow-up list ignored. Note what it still requires: a contract price does NOT excuse a short confirm or a late promise.
   d. QUANTITY SHORT. The acknowledged quantity is LESS than the ordered quantity. Answer 'qty_short'. This sits above price and date because being told you will not get what you ordered changes what those two are about.
   e. PRICE MISMATCH. The acknowledged unit price differs from the ORDERED unit price by MORE than 1 minor unit(s). Answer 'price_mismatch'.
   f. DATE SLIP. The promised date is MORE than 2 day(s) later than the requested date. Answer 'date_slip'. An earlier promise is never a slip.
   g. Anything that survives all six is 'confirmed'.
9. THE ITEM REFERENCE DECIDES NOTHING. This rulebook does not compare `ack_item_ref` with `item_ref`, and you must not either. A line whose quantity, price and date all match is 'confirmed' even where the two references differ. Extract both faithfully and let the difference stand.
10. HOW LONG THE ACKNOWLEDGMENT TOOK TO ARRIVE DECIDES NOTHING EITHER. There is no window in these gates. A line answered late is judged on its values exactly like one answered the same day.
11. THE BUYER NOTE IS A FIELD TO COPY, NOT EVIDENCE ABOUT ANY LINE. A note that sounds settled does NOT clear a mismatch, and a note that sounds worried does NOT create one. The order's own numbers decide; the note is one person's remark and may disagree with them.
12. Copy every reference, every quantity and every price verbatim, exactly as printed. Every date is yyyy-mm-dd.
13. Use the exact allowed value for a field that lists them, and return every field named in the schema for every line, even when the answer is null.

PURCHASE ACKNOWLEDGMENT RULEBOOK PAR-1 (the authority for `reconciliation`; this is an ILLUSTRATIVE rulebook shipped with this kit, not a law, a regulation, an EDI guideline or anybody's procurement policy)

THE THREE FACTS COMPARED
  quantity, unit price, delivery date
  The three facts a purchase order line states and a supplier acknowledgment answers. THE WHOLE KIT IS THE COMPARISON BETWEEN THE TWO SIDES OF EACH. A supplier who answered at all is not the finding; a supplier whose answer differs from the order, or who answered nothing, is.

PRICE TOLERANCE
  1 minor unit(s). A supplier may acknowledge a unit price one minor unit either side of the ordered price -- a rounding artefact of a converted or per-thousand price -- and that is not a mismatch.

SCHEDULE BUFFER
  2 day(s). A promise up to this many days later than the requested date is inside the schedule buffer and is not a slip. An EARLIER promise is never a finding -- a supplier offering to deliver ahead of the request is not an exception, and a checker that flags it is a checker nobody reads.

THE ACKNOWLEDGMENT WINDOW -- AND WHY IT DECIDES NOTHING
  3 day(s), ASSUMED: NOT SUPPLIED BY ANY OPERATOR. No acknowledgment window exists for this use case; one had to be set to compute anything at all, so 3 days was set FOR THIS KIT and marked as an assumption rather than published as a policy.
  THE WINDOW DECIDES NOTHING AND THAT IS THE POINT. The kit computes how long an order sat before the acknowledgment arrived -- or, where none arrived, how long it has been sitting at the moment the extract was taken -- and reports it. It does NOT use it to reach a status: `unacknowledged` fires on the ABSENCE of an acknowledgment line, never on its age. While `assumed` is true, every window reading renders `unconfirmed` and is never a finding. Supply a real window and the flag becomes answerable; until then a number nobody agreed to must not sit underneath a flag somebody acts on.

THE SEVEN GATES, IN ORDER. STOP AT THE FIRST THAT FIRES.
  1. UNACKNOWLEDGED:        The Supplier Acknowledgment section carries no line for this order line at all.
     why: SILENCE IS THE ANSWER, AND IT IS THE ONE AN INCOMING-DOCUMENT MATCHER CANNOT GIVE. A check that walks the acknowledgments it received and compares each to its order line has no row to walk for a line nobody answered, so the line is invisible to it rather than clean. This gate is first because there is nothing to compare, and because it is the population the audit question actually asks about.
  2. NOT_DETERMINABLE:      An acknowledgment line exists for this order line, and at least one of the acknowledged quantity, unit price or promised date is not recorded on it -- or the order line itself does not state one of its own three values.
     why: On a follow-up queue the expensive mistake is a confident wrong 'fine'. A line answered with a value missing is a record somebody has to open -- not a line that can be cleared, and not a line anybody can chase the supplier about until it is read.
  3. AUTHORISED_VARIANCE:   A contracted unit price is on file for this item for this period; the acknowledged unit price equals it to the minor unit; the acknowledged quantity equals the ordered quantity; and the promised date is not later than requested beyond the buffer.
     why: This is the difference that is SUPPOSED to be there. An order raised at a stale price against a contract that has since moved will come back acknowledged at the contracted price, correctly, and a queue that reports every one of them buries the real mismatches underneath them. ⚑ NOTE WHAT IT REQUIRES: the quantity and the date must still be right. A contract price does not excuse a short confirm.
  4. QTY_SHORT:             The acknowledged quantity is LESS than the ordered quantity.
     why: The supplier is telling you that you will not get what you ordered. That is a supply decision rather than a paperwork one, and it sits above the price and date gates because it changes what those two are about: a price per unit on two thirds of a line is a different conversation from a price per unit on all of it. ⚠︎ AN ACKNOWLEDGED QUANTITY GREATER THAN ORDERED IS OUT OF SCOPE HERE and would fall through to the gates below. The corpus contains none and evals/check_labels.py asserts that, so nothing is silently mis-scored -- but a real acknowledgment file contains them and this rulebook does not answer for them.
  5. PRICE_MISMATCH:        The acknowledged unit price differs from the ordered unit price by more than the price tolerance, and no contracted price on file explains it.
     why: The line will not three-way match at invoice. It is named separately from a date slip because it goes to a different desk: somebody has to decide whether to accept the price, and no expediting call answers that.
  6. DATE_SLIP:             The promised date is later than the requested date by more than the buffer.
     why: The line is confirmed and it is late. That is a schedule problem a planner absorbs or escalates, and it is deliberately NOT the same finding as a price change -- routing both into one queue is how the one that needs a decision gets buried under the one that needs a phone call.
  7. CONFIRMED:             Everything else: the supplier acknowledged this line at the ordered quantity, at a price within tolerance, on or before the requested date plus the buffer.
     why: Nothing needs doing. ⚠︎ AND NOTE WHAT THIS DOES NOT SAY. It does not say the supplier acknowledged the RIGHT ITEM -- see `item_identity_note`.

THE SEVEN STATUSES
  confirmed, authorised_variance, date_slip, price_mismatch, qty_short, unacknowledged, not_determinable

WHAT PUTS A LINE ON THE FOLLOW-UP LIST
  unacknowledged, qty_short, price_mismatch, date_slip -- The four statuses that put a line on somebody's follow-up list today. `confirmed` and `authorised_variance` are the two answers that mean nothing needs doing. `not_determinable` means the RECORD needs opening rather than the SUPPLIER needing chasing, and it is deliberately not on the exception list for that reason -- it is reported and counted beside the list instead.

THE PRICE GAP
  The absolute difference between the acknowledged unit price and the ORDERED unit price, in minor units. Not against the contracted price -- against the order, because the question is what the buyer asked for and what the supplier answered. Null where either price is not recorded, and null where the supplier acknowledged nothing.

THE SLIP IN DAYS
  The promised date minus the requested date, in whole days, SIGNED. Negative means the supplier promised earlier than requested, which is not a finding. Null where either date is not recorded, and null where the supplier acknowledged nothing.

WHAT THE RULEBOOK DOES NOT LOOK AT
  ⚑ THE ONE THING THIS RULEBOOK DOES NOT LOOK AT. The acknowledgment states the supplier's own item reference beside each line, and the gates above never compare it to the item the order line names. An acknowledgment that confirms the quantity, the price and the date against a DIFFERENT item is `confirmed` here -- and it will not three-way match at receipt. The kit extracts both references, reports them side by side, and counts the lines where they differ; it does not act on them, because adding a gate would change every published figure on this page. The count is printed beside every follow-up list rather than folded into 'nothing to do'.

WHAT THIS TOOL WILL NOT DO
  - It never accepts, approves or confirms an acknowledgment.
  - It never writes to, amends, cancels or re-issues a purchase order.
  - It never contacts a supplier, sends a chase, or opens a message of any kind.
  - It never decides whether an acknowledged price or quantity should be accepted.
  - It never uses the assumed acknowledgment window to reach a status.

Extract these fields:
ORDER-LEVEL FIELDS -- one value each, for the whole purchase order
- po_ref (string) -- the purchase order reference, verbatim from the Purchase Order section
- order_placed_on (date) -- the date the order was placed, from the Order Placed On section, as yyyy-mm-dd
- supplier_ref (string) -- the supplier reference, verbatim from the Supplier Reference section -- the short code only, not the trading name after it
- rulebook_id (string) -- the identifier of the acknowledgment rulebook this order is checked under, verbatim -- the short code only, not the sentence after it
- acknowledgment_received_on (date) -- the date the supplier's acknowledgment arrived, from the Acknowledgment Receipt section, as yyyy-mm-dd. Return null where that section says no acknowledgment was received for this order. Do NOT infer it from any promised date
- extract_taken_on (date) -- the date this extract was taken, from the Acknowledgment Receipt section, as yyyy-mm-dd. It is stated on every order, including the ones with no acknowledgment
- buyer_note (string) -- the buying desk's own free-text remark from the Buyer Notes section, copied verbatim. It is a field to copy and it is NOT evidence about whether any line is right

PER-LINE FIELDS -- one object per row of Order Lines
- line_ref (string) -- the line reference as the Order Lines section states it, e.g. L040. Return one object per row of THAT section, in the order it lists them, and no others. The Supplier Acknowledgment section carries rows for line references this order does not have -- those are not yours
- item_ref (string) -- the item reference the ORDER LINE names, e.g. MAT-40185, read from the Order Lines row. This is what the buyer ordered
- ordered_qty (integer) -- the quantity ordered, read after 'qty' on the Order Lines row. A whole number, digits only, no thousands separator
- ordered_unit_price (decimal) -- the unit price the order was raised at, read after 'at' on the Order Lines row. Copy the digits exactly as printed
- requested_date (date) -- the delivery date the order asks for, read after 'wanted' on the Order Lines row, as yyyy-mm-dd
- ack_item_ref (string) -- the item reference the SUPPLIER'S ACKNOWLEDGMENT names for this line, read after 'item' on the Supplier Acknowledgment row. Return null where that section carries no row for this line reference at all. ⚠︎ COPY WHAT THE ACKNOWLEDGMENT SAYS. It is not always the same reference the order line names, and correcting it to match would destroy the only evidence of a substitution
- ack_qty (integer) -- the quantity the supplier acknowledged, read after 'ack qty' on the Supplier Acknowledgment row. Return null where that section carries no row for this line, and null where the row reads 'ack qty -- not stated'
- ack_unit_price (decimal) -- the unit price the supplier acknowledged, read after 'at' on the Supplier Acknowledgment row. Return null where that section carries no row for this line, and null where the row reads 'at -- not stated'
- ack_promised_date (date) -- the delivery date the supplier promised, read after 'promised' on the Supplier Acknowledgment row, as yyyy-mm-dd. Return null where that section carries no row for this line, and null where the row reads 'promised -- not stated'
- contract_unit_price (decimal) -- the contracted unit price the Contract Terms On File section records for THIS ORDER LINE'S item reference. Return null where that section carries no row for it. That section also carries rows for items this order does not buy
- price_gap_minor (derived) -- the ABSOLUTE difference, in MINOR UNITS (hundredths), between the acknowledged unit price and the ORDERED unit price. NO ORDER STATES IT; work it out. An integer, so a difference of 0.07 is 7. Return 0 where the two are equal, and null where either price is not recorded or the supplier acknowledged nothing. It is measured against the ORDER, not against any contracted price
- slip_days (derived) -- the promised date MINUS the requested date, in whole days, SIGNED. NO ORDER STATES IT; work it out. Negative where the supplier promised EARLIER than requested, 0 where they match. Return null where either date is not recorded or the supplier acknowledged nothing
- reconciliation (enum) one of: confirmed, authorised_variance, date_slip, price_mismatch, qty_short, unacknowledged, not_determinable -- this line's status under the shipped rulebook. Work through seven gates IN ORDER and stop at the first that fires. (1) UNACKNOWLEDGED: the Supplier Acknowledgment section carries no row for this line reference at all. (2) NOT DETERMINABLE: a row exists and the acknowledged quantity, price or promised date is not recorded on it, or the order line itself is missing one of its own three values. (3) AUTHORISED VARIANCE: a contracted unit price is on file for this line's item, the acknowledged price equals it, the acknowledged quantity equals the ordered quantity, and the promise is not later than requested beyond the buffer. (4) QTY SHORT: the acknowledged quantity is less than the ordered quantity. (5) PRICE MISMATCH: the acknowledged price differs from the ORDERED price by more than the tolerance. (6) DATE SLIP: the promised date is later than the requested date by more than the buffer -- an EARLIER promise is never a slip. (7) Anything that survives all six is CONFIRMED. Never decide it from the Buyer Notes remark, and never from how long the acknowledgment took to arrive

Return a JSON object of exactly this shape:
{"po_ref": ..., "order_placed_on": ..., "supplier_ref": ..., "rulebook_id": ..., "acknowledgment_received_on": ..., "extract_taken_on": ..., "buyer_note": ..., "lines": [{"line_ref": ..., "item_ref": ..., "ordered_qty": ..., "ordered_unit_price": ..., "requested_date": ..., "ack_item_ref": ..., "ack_qty": ..., "ack_unit_price": ..., "ack_promised_date": ..., "contract_unit_price": ..., "price_gap_minor": ..., "slip_days": ..., "reconciliation": ...}, ...]}
Use null for any field the record does not state.

PURCHASE ORDER AND ACKNOWLEDGMENT
---------------------------------
Purchase Order
--------------
PO-3250-GK

Order Placed On
---------------
2026-02-24

Supplier Reference
------------------
SUP-9056 -- Brightmoor Industrial

Acknowledgment Rulebook
-----------------------
PAR-1 -- illustrative, shipped with this kit; reproduces no real procurement rule

Order Lines
-----------
quantities are in the ordering unit of measure; prices are the trading currency's major unit
L010    MAT-44135     Tap, M8 x 1.25, spiral flute               qty 40       at 7.59       wanted 2026-04-10
L020    MAT-18572     Proximity sensor, inductive, M12           qty 15       at 55.92      wanted 2026-04-27
L030    MAT-94320     Timing belt, 8mm pitch, 60 teeth           qty 26       at 20.82      wanted 2026-04-01
L040    MAT-66563     Abrasive flap disc, 115mm, 60 grit         qty 125      at 1.29       wanted 2026-04-05
L050    MAT-52255     Socket cap screw M6 x 25                   qty 1190     at 0.28       wanted 2026-05-10
L060    MAT-23554     Solenoid valve, 5/2, 1/4 inch              qty 14       at 68.05      wanted 2026-03-27
L070    MAT-78248     Carbide insert, CNMG 120408                qty 210      at 4.93       wanted 2026-04-01
L080    MAT-11952     Anti-vibration mount, 40 shore             qty 115      at 7.75       wanted 2026-05-24

Supplier Acknowledgment
-----------------------
L250    item MAT-99225     ack qty 185             at 9.76                 promised 2026-04-06
L190    item MAT-24911     ack qty 32              at 51.16                promised 2026-04-14
L060    item MAT-23554     ack qty 14              at 68.05                promised 2026-04-08
L040    item MAT-66563     ack qty 125             at 1.29                 promised -- not stated
L070    item MAT-78248     ack qty 210             at 5.60                 promised 2026-04-01
L050    item MAT-52255     ack qty 1190            at 0.32                 promised 2026-05-10
L010    item MAT-44135     ack qty 40              at 7.59                 promised 2026-04-06
L280    item MAT-21606     ack qty 1990            at 0.36                 promised 2026-05-06
L080    item MAT-11952     ack qty 115             at 9.25                 promised 2026-05-24

Acknowledgment Receipt
----------------------
acknowledgment received 2026-02-27; extract taken 2026-03-08

Contract Terms On File
----------------------
MAT-52255     contracted unit price 0.32 for this period
MAT-14331     contracted unit price 0.58 for this period

Buyer Notes
-----------
Flagged this one for a second pass -- the acknowledgment did not read right to me.

Raw responseThe raw response

{"po_ref":"PO-3250-GK","order_placed_on":"2026-02-24","supplier_ref":"SUP-9056","rulebook_id":"PAR-1","acknowledgment_received_on":"2026-02-27","extract_taken_on":"2026-03-08","buyer_note":"Flagged this one for a second pass -- the acknowledgment did not read right to me.","lines":[{"line_ref":"L010","item_ref":"MAT-44135","ordered_qty":40,"ordered_unit_price":7.59,"requested_date":"2026-04-10","ack_item_ref":"MAT-44135","ack_qty":40,"ack_unit_price":7.59,"ack_promised_date":"2026-04-06","contract_unit_price":null,"price_gap_minor":0,"slip_days":-4,"reconciliation":"confirmed"},{"line_ref":"L020","item_ref":"MAT-18572","ordered_qty":15,"ordered_unit_price":55.92,"requested_date":"2026-04-27","ack_item_ref":null,"ack_qty":null,"ack_unit_price":null,"ack_promised_date":null,"contract_unit_price":null,"price_gap_minor":null,"slip_days":null,"reconciliation":"unacknowledged"},{"line_ref":"L030","item_ref":"MAT-94320","ordered_qty":26,"ordered_unit_price":20.82,"requested_date":"2026-04-01","ack_item_ref":null,"ack_qty":null,"ack_unit_price":null,"ack_promised_date":null,"contract_unit_price":null,"price_gap_minor":null,"slip_days":null,"reconciliation":"unacknowledged"},{"line_ref":"L040","item_ref":"MAT-66563","ordered_qty":125,"ordered_unit_price":1.29,"requested_date":"2026-04-05","ack_item_ref":"MAT-66563","ack_qty":125,"ack_unit_price":1.29,"ack_promised_date":null,"contract_unit_price":null,"price_gap_minor":0,"slip_days":null,"reconciliation":"not_determinable"},{"line_ref":"L050","item_ref":"MAT-52255","ordered_qty":1190,"ordered_unit_price":0.28,"requested_date":"2026-05-10","ack_item_ref":"MAT-52255","ack_qty":1190,"ack_unit_price":0.32,"ack_promised_date":"2026-05-10","contract_unit_price":0.32,"price_gap_minor":4,"slip_days":0,"reconciliation":"authorised_variance"},{"line_ref":"L060","item_ref":"MAT-23554","ordered_qty":14,"ordered_unit_price":68.05,"requested_date":"2026-03-27","ack_item_ref":"MAT-23554","ack_qty":14,"ack_unit_price":68.05,"ack_promised_date":"2026-04-08","contract_unit_price":null,"price_gap_minor":0,"slip_days":12,"reconciliation":"date_slip"},{"line_ref":"L070","item_ref":"MAT-78248","ordered_qty":210,"ordered_unit_price":4.93,"requested_date":"2026-04-01","ack_item_ref":"MAT-78248","ack_qty":210,"ack_unit_price":5.60,"ack_promised_date":"2026-04-01","contract_unit_price":null,"price_gap_minor":67,"slip_days":0,"reconciliation":"price_mismatch"},{"line_ref":"L080","item_ref":"MAT-11952","ordered_qty":115,"ordered_unit_price":7.75,"requested_date":"2026-05-24","ack_item_ref":"MAT-11952","ack_qty":115,"ack_unit_price":9.25,"ack_promised_date":"2026-05-24","contract_unit_price":null,"price_gap_minor":150,"slip_days":0,"reconciliation":"price_mismatch"}]}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch the purchase-order lines your supplier never answered — 50 PO acknowledgments. One model answered, and every answer was then graded Six different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid -- the real spend is in the shared call ledger, not on this page.

The gradersSix ways to grade

The floor is scored by the SAME judge, over the same gold, on the same 50 records. A floor scored by a different scorer from the run it is compared against is not a floor, it is a second kit.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same order, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The free incoming-document matcher (evals/baseline.py), the gold gate (evals/check_labels.py), the stub (--stub) and the scorer (evals/judge.py) are all pure code and cost $0.00 to run against any result set. The figure above is the ONE scored run's own token counts (50 records) priced on the published Google Gemini 3 Flash card. ⚑ THE WHOLE KIT COST 62 LIVE CALLS TO BUILD, NOT 50. The scored run is 50; the calibration that set the ceiling is 4 more and projects to $0.0874; the prompt-split measurement is 4 at max_tokens=1; the worked example is 1; and the answered screenshot is 3, because the first two were defects only opening the image could find. Every one of those is a real call against a real key and none of them is in the figure above, which prices the scored run alone.

Cost driversWhat actually moves the bill

• PROVIDER-SIDE REASONING, by a distance. 137225 of 177706 output tokens (77.22 pct) never reach the reply text and are billed in full. The JSON roster itself is the other 22.78 pct.
• THE FIXED PREFIX, which is the same on every call. 4747 of 5549 input tokens (85.5 pct) are the system prompt, the rulebook and the field schema; the order record itself is 802. Nothing here caches it.
• THE RULEBOOK, at 1784 tokens -- 32.1 pct of every call and the single largest part. It is the part that grows with your POLICY rather than with your DOCUMENT: seven gates, two tolerances and one assumed window render to that, and a real supplier agreement renders to more, into every one of thousands of calls, while the orders stay the same size.
• THE LINE COUNT, which drives the output. The reply is one thirteen-field object per line, so a nine-line order costs meaningfully more than a four-line one -- and the corpus averages 6.2.

Your volumeWhat it costs at your volume

Linear in orders and NOT linear in order size. Ten times the orders is ten times the bill: each is one independent call with an identical prefix. But ten times the LINES PER ORDER is worse than ten times the cost, because the reply is a roster -- output grows with the line count and the reasoning pass grows with it too, and past the ceiling the whole order is lost rather than the extra lines. ⚠︎ AND THE MEASURED SPREAD SAYS THE LINE COUNT IS NOT THE ONLY VARIABLE. This run's replies ran 815 to 12582 output tokens -- a 15.44x spread -- over documents that are 1533 to 3118 bytes, a 103.4 pct spread. Anyone sizing this from document length will be wrong by an order of magnitude in either direction.

Model choiceWhy this model

It is the tier the shared key on this machine is pointed at, and one tier is what this run bought. The kit is provider-agnostic -- src/adapters reaches an OpenAI-compatible endpoint or Anthropic's Messages API over urllib -- so 'why this model' is honestly 'it is the one configured', and the honest consequence is that this page contains no tier comparison at all. The rows under cost_projection are PROJECTIONS of this run's measured tokens onto twelve published cards, not runs.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 purchase orders, one completion call each, one tier. The calibration, the worked-example capture, the prompt-split calibration and the screenshots are separate and are NOT in these totals -- see Cost.cost_of_evaluation_usd, which counts them.

The codeEvery file, and what it is for

10 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pyOrder record segmenter

cuts one order record into addressable sections on its underlined headings; falls back to one whole-document segment when it finds none

# Cut a purchase-order record into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pySection selector — a swap seam

maps each of the 20 fields to the sections that can carry it, and subtracts the two sections no field maps to -- unconditionally, including on the fallback path

You change it to: which sections reach the provider at all. NEVER_SENT is the privacy line; _fallback subtracts it unconditionally so an absent section cannot widen the wire.

# Pick which sections of a purchase-order record are sent. Pure code -- the last deterministic step

PO = "Purchase Order"
PLACED = "Order Placed On"
SUPPLIER = "Supplier Reference"
RULEBOOK = "Acknowledgment Rulebook"
LINES = "Order Lines"
ACK = "Supplier Acknowledgment"
RECEIPT = "Acknowledgment Receipt"
CONTRACT = "Contract Terms On File"
NOTES = "Buyer Notes"

src/prompt.pyPrompt assembler

one call per order: the system prompt, the rulebook rendered from its own JSON, the field schema, and the mapped sections

# Assemble the po-ack prompt. ONE prompt per purchase order, every line on it in one reply.

VERDICTS = ("confirmed", "authorised_variance", "date_slip", "price_mismatch",
VERDICT_MEANINGS = {
SYSTEM = (
def rulebook_block(rb=None):
RULEBOOK_TEXT = rulebook_block()
def field_schema(fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/adapters/__init__.pyProvider adapter

urllib against an OpenAI-compatible endpoint or Anthropic's Messages API, with backoff over both HTTP and transport failures

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/rulebook.pyThe acknowledgment rulebook

the seven gates, in order, as pure code over integer thousandths and whole days -- one definition read by the corpus generator, the prompt and the scorer

# THE PURCHASE-ACKNOWLEDGMENT RULEBOOK, loaded from data/rulebook.json. Pure code, no model, no

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
STATUSES = ("confirmed", "authorised_variance", "date_slip", "price_mismatch",
EXCEPTIONS = ("unacknowledged", "qty_short", "price_mismatch", "date_slip")
NOTHING_TO_DO = ("confirmed", "authorised_variance")
DECISION_STATUSES = ("price_mismatch", "qty_short", "not_determinable")
COMPLETENESS_STATES = ("complete", "partial", "none")
def load(path=RULEBOOK_PATH):
RB = load()

src/extract.pyReconciler — a swap seam

drives the whole path and attaches a span to every spannable value, searched inside the section the field is mapped to and on the line's own row before it ever widens

You change it to: MAX_TOKENS. A cut reply costs a whole order and produces no follow-up list at all, which looks exactly like an order with nothing wrong.

# Reconcile one purchase order against its acknowledgment: segment, select, prompt, ONE model

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 48000
def load_fields():
def line_key(fields):
def scored_line_fields(fields):
def load_doc(po_id):
def documents():

src/extract.pyOrder routing rule — a swap seam

the pure-code business condition -- does this order need somebody to decide, or somebody to chase

You change it to: MAX_TOKENS. A cut reply costs a whole order and produces no follow-up list at all, which looks exactly like an order with nothing wrong.

# Reconcile one purchase order against its acknowledgment: segment, select, prompt, ONE model

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 48000
def load_fields():
def line_key(fields):
def scored_line_fields(fields):
def load_doc(po_id):
def documents():

evals/baseline.pyThe free floor

the incoming-document matcher: every acknowledgment row against its order line, blind to a line nobody answered by construction

# A free, rules-and-regex reconciliation. No model, no key, no spend -- scored by the same judge as

NOT_STATED = "-- not stated"
def _section(text, name):
def _rows(text, name):
def _cols(line):
def _roster(text):
def _acks(text):
def _cells(values, line_fields):
def extract(text, fields):

evals/judge.pyScorer

six graders, pure code, no LLM judge

# Score a po-ack run. PURE CODE -- gold is exact and every answer is one value, so `==` (with light

def norm(v):
def equal(field, got, want):
def _rows_by_ref(rows, key="line_ref"):
def score(fields, records, golds):
def _matrix(rows, positive):
NOTHING_TO_DO = RB.NOTHING_TO_DO
def is_unacknowledged(gp):
def score_statuses(fields, records, golds):
def _same_int(got, want):

src/app.pyLocal UI

one order, one call, on 127.0.0.1:9002; renders with no key

# The minimal local UI. Standard library only -- python -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "9002"))
class H(BaseHTTPRequestHandler):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 5425 input and 3554 output tokens per order, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's order records are entirely synthetic (tools/build_corpus.py, seed 20260823): no real manufacturer, supplier, part, brand, person, price or currency exists in the corpus, and no third-party data was fetched. Each record does carry an invented BUYER CONTACT block -- a named buyer, a direct telephone extension and an internal requisition account -- and a PREPARED BY line, and neither ever leaves the machine: no field maps to either section, so the union src/select.py sends excludes both on every one of the 50 documents, on the fallback path, and for a field name no hint knows. evals/check_labels.py drives all four paths directly rather than inferring the guard from one sample. The kit performs no write of any kind. It has no database, no queue, no scheduler and no state; the only outbound call in the whole repository is the single completion in src/adapters. It cannot accept an acknowledgment, amend a purchase order or contact a supplier because no code path does any of those things.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the local app redacts the key and the base URL out of the message before it reaches a browser.

The experimentWe did not attack it -- and the privacy boundary is guarded by a check, not by luck

Three boundaries hold. Two were confirmed by reading the code: no code path accepts an acknowledgment, amends an order, contacts a supplier or writes anything at all, and the rulebook reads six values and two tolerances and nothing else. The third was confirmed by driving it: evals/check_labels.py calls src/select.py's fallback DIRECTLY on all 50 documents, plus an unmapped field name, plus all 20 mapped fields, and asserts that Buyer Contact and Prepared By appear on none of them. That path never fires on this corpus -- every document carries every mapped section -- so a guard tested only by running the corpus would be untested and would still read as passing. The fourth boundary was not tested at all. Nothing was crafted, nothing was injected, and 0 trials is what this page publishes. Confirmed by reading the code and by a pre-flight assertion that drives the fallback path directly, not by an attack run, on 2026-08-23 -- no attack run was fired; see redteam.why.

Three boundaries hold and the second is the one worth reading, because it holds by a check rather than by luck: the fallback path that would leak the buyer contact never fires on this corpus, so the gold gate drives it directly on all 50 documents instead of trusting it. The fourth is unmeasured and is stated as unmeasured. ⚑ THE FIRST IS THE ATLAS ROW'S OWN GUARDRAIL -- decision-free by design, no accept, no PO update, no supplier contact in any output -- and it holds structurally rather than by instruction: there is nothing in the repository that could perform any of the three.

The result0 attack trials. Three of four boundaries checked here hold on evidence -- and the privacy one holds because a gate drives a code path this corpus never reaches, not because the path happens never to fire. The fourth, whether text inside a supplier's returned acknowledgment could move a verdict, is UNMEASURED and is named as unmeasured.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260823), so no text in it came from an outside party and there was nothing adversarial to resist. What the corpus DOES plant is confusable plain record-keeping, and the run's result on it is measured: 100.0 pct on the 20 records whose buyer note contradicts their own numbers against 100.0 pct on the 30 where it agrees. That is a real number about a real trap and it is not a resistance rate.

HonestyWhat this does not prove

• Whether an indirect prompt injection in a supplier's returned acknowledgment or a Buyer Notes line could move a call. No attack run exists, and the make-it-read-confirmed version is the one that matters: a line cleared is a line nobody looks at again.
• Whether the kit behaves safely against a hostile provider -- a response body crafted to break the roster parsing in src/prompt.py::parse, or to return an enormous payload. The parser fails closed to an empty roster, which the harness records as a failed order rather than an empty one, but no adversarial body was ever sent.
• Whether a real order extract would carry anything sensitive this kit mishandles beyond the buyer contact block. The corpus has no personal data by construction; a real one carries negotiated prices, supplier terms and a production schedule, and only a reader of a real extract can say what else must never reach a provider.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_buyer_decision fires when the order carries at least one line whose computed status is `price_mismatch`, at least one `qty_short`, or at least one `not_determinable`. All three come out of src/rulebook.py over the values the model returned; the flag is a predicate over them and never sees a document. An `unacknowledged` line is a CHASE and a `date_slip` is a reschedule -- neither needs anybody to decide anything, and both are deliberately excluded.

src/extract.py::compute(), called by extract() on every record, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by the same function. It returns True, False, or None when a status the rule needs is missing -- an unknown is not a pass.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT an acceptance, an order amendment, a chase or a decision about what any price or quantity should be. Nothing here changes anything. reconciliation is a finding with the rulebook's reasoning attached, and the flag is a routing hint over those findings.
• It is NOT a check that the follow-up list is COMPLETE, and the corpus is the demonstration. The rule reads the reply; if the reply misses a line, the rule is silent about it and the flag is confidently wrong in the safe-looking direction. Roster coverage is published separately for exactly that reason -- 308 of 308 on this run.
• It is NOT anybody's escalation policy. “A price or a quantity the supplier will not hold needs a decision; silence needs a chase; a slip needs a reschedule” is this kit's own simplification, invented for it. No manufacturer's procedure, contract or regulation was consulted and none is reproduced.
• ⚠︎ AND IT NAMES NO OWNER, BECAUSE NOBODY SUPPLIED ONE. Who actually sends a supplier follow-up -- the buyer directly, or a shared procurement inbox -- is an OPEN QUESTION on this use case. The flag says what KIND of work an order needs and stops there; a flag with no owner is a flag somebody still has to route by hand.
• It is NOT a judgement about the rulebook. The flag reads the statuses the rulebook produced; if the rulebook is wrong, the flag is confidently wrong with it. The shipped rulebook is an invention and nobody qualified has reviewed it.
• ⚠︎ AND IT IS NOT AN ITEM-IDENTITY CHECK. A line acknowledged at the right quantity, the right price and the right date against a DIFFERENT item is confirmed, so it contributes nothing to this flag -- and 12 such lines in this corpus are downstream exceptions. The count is reported beside the list and the flag does not read it.

WatchedWhat is watched, and why that one

The kit measures 29 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 22 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 50, extraction_cells 4046, failures 0, refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

2 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 2 runs measured the same system — they differ on provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 2 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/extract.py, on any change to data/rulebook.json (which moves the statuses the flag reads), on any change to the corpus composition (which moves how many orders it can fire on), and on any run against a different provider or model. Its precision and recall are both consequences of the line calls, so a run whose line accuracy moves invalidates this block whether or not the predicate changed.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the rulebook is a JSON file read with json.load, the model is reached over urllib, and the UI is http.server plus one page of vanilla JS. A forker clones this and runs it; there is nothing to install and nothing to understand before touching the part they came for.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

One path per record -- segment, select, prompt, one call, parse, span, the rule -- with no branch, no loop and no agent. What the reply contains is a ROSTER rather than one object, and the roster is the ORDER's lines rather than the acknowledgment's rows, which is the single structural decision the whole kit turns on: enumerate the acknowledgment and silence becomes invisible.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain -- the JSON extraction from a fenced reply, the roster parsing, the backoff policy, the section regex, the money arithmetic and the date arithmetic. Each is small; together they are a few hundred lines nobody else is patching.
• No framework means no framework's ecosystem -- no tracing, no eval harness beyond the one in evals/, no prompt registry, no structured-output enforcement. The reply is a JSON object parsed with json.loads and one keyword; a function-calling or structured-output API would make the shape a provider guarantee rather than an instruction, and would plausibly have made the trailing-zero question moot by declaring the price fields STRING.
• The corpus generator, the gold gate and the scorer are three separate programs that must agree, and only convention holds them together -- one shared import of src/rulebook.py and a set of asserts. Nothing enforces that a fourth reader could not be added without one.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-po-ack on the fast tier, 2026-08-23. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 1 earlier run on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 1 run on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT (this repository's own licence). Nothing was fetched and nothing is licensed from anybody: no real manufacturer, supplier, part, brand, person or currency is named anywhere in the kit. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-(line, field) exact match, light normalisation

Does the model's value for each of the 12 per-line fields match gold after trimming whitespace and punctuation, plus 7 order-level cells per record? THE DENOMINATOR IS GOLD'S LINES, NOT THE RUN'S: a line the reply never mentioned is 12 wrong cells and a status of unanswered, never quietly excluded -- and on this kit the lines most likely to be dropped are the ones with no acknowledgment row to copy from, which are exactly the rows the kit exists to surface.

MethodHow the test was run

evals/judge.py, in-process, no key and no model. The same function evals/run.py calls after every run, and the same logic the free floor is scored by -- a floor and a real run scored by two different scorers is not a floor, it is a second kit.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

Read what the FLOOR does with this row rather than what the model does. The free incoming-document matcher emits NO ROW for it at all -- not confirmed, not an exception, nothing -- so the judge records it as unanswered and it reaches nobody's list. That is not a bug in the floor; it is what an acknowledgment matcher IS. This row and the 37 like it are the entire reason this kit was built, and the model path caught 38 of 38 of them. ⚑ AND NOTE WHAT THE WINDOW DOES HERE, WHICH IS NOTHING. The order sat 3 day(s) before the acknowledgment arrived. The kit prints that and renders the verdict unconfirmed, because the 3-day window it would be measured against is an assumption nobody supplied. The status on this line came from the ABSENCE of a row, not from its age.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, derived by running src/rulebook.py over the same values each record states; evals/check_labels.py re-runs the rule over all 308 line rows and asserts every hard case has a floor before any run may spend. The ONE label that is not derived is downstream_exception, which is planted per bucket in exact counts and appears in no corpus file.

How tight can the band be? No tolerance band in the SCORER. Exact match after trimming whitespace and punctuation for cells; integer comparison for the price gap and the slip; a closed set of seven values for the status. The two tolerances that exist -- 1 minor unit and 2 days -- are in the RULEBOOK, where they are part of the question rather than part of the marking.

Cadence: Re-score on any change to data/rulebook.json, data/fields.json, src/prompt.py or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe reconciliation call, seven-way, over the model's own values

Does src/rulebook.py, run over the values the MODEL returned, land on gold's status? This is the deciding half and it is not a tautology: a misread quantity or a misread date flips a status, so it measures the whole pipeline rather than the rule.

MethodHow the test was run

evals/judge.py, in-process, no key and no model. The same function evals/run.py calls after every run, and the same logic the free floor is scored by -- a floor and a real run scored by two different scorers is not a floor, it is a second kit.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

Read what the FLOOR does with this row rather than what the model does. The free incoming-document matcher emits NO ROW for it at all -- not confirmed, not an exception, nothing -- so the judge records it as unanswered and it reaches nobody's list. That is not a bug in the floor; it is what an acknowledgment matcher IS. This row and the 37 like it are the entire reason this kit was built, and the model path caught 38 of 38 of them. ⚑ AND NOTE WHAT THE WINDOW DOES HERE, WHICH IS NOTHING. The order sat 3 day(s) before the acknowledgment arrived. The kit prints that and renders the verdict unconfirmed, because the 3-day window it would be measured against is an assumption nobody supplied. The status on this line came from the ABSENCE of a row, not from its age.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, derived by running src/rulebook.py over the same values each record states; evals/check_labels.py re-runs the rule over all 308 line rows and asserts every hard case has a floor before any run may spend. The ONE label that is not derived is downstream_exception, which is planted per bucket in exact counts and appears in no corpus file.

How tight can the band be? No tolerance band in the SCORER. Exact match after trimming whitespace and punctuation for cells; integer comparison for the price gap and the slip; a closed set of seven values for the status. The two tolerances that exist -- 1 minor unit and 2 days -- are in the RULEBOOK, where they are part of the question rather than part of the marking.

Cadence: Re-score on any change to data/rulebook.json, data/fields.json, src/prompt.py or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe derived arithmetic, the model's own, which decides nothing

Two integers no record states: price_gap_minor, the absolute difference between the acknowledged and the ORDERED unit price in minor units, and slip_days, the promised date minus the requested date in whole days, SIGNED. The kit publishes the CODE's figures; the model is asked for its own and both are measured beside them. Both are integers on purpose -- an integer cannot lose a trailing zero to a JSON parser.

MethodHow the test was run

evals/judge.py, in-process, no key and no model. The same function evals/run.py calls after every run, and the same logic the free floor is scored by -- a floor and a real run scored by two different scorers is not a floor, it is a second kit.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

Read what the FLOOR does with this row rather than what the model does. The free incoming-document matcher emits NO ROW for it at all -- not confirmed, not an exception, nothing -- so the judge records it as unanswered and it reaches nobody's list. That is not a bug in the floor; it is what an acknowledgment matcher IS. This row and the 37 like it are the entire reason this kit was built, and the model path caught 38 of 38 of them. ⚑ AND NOTE WHAT THE WINDOW DOES HERE, WHICH IS NOTHING. The order sat 3 day(s) before the acknowledgment arrived. The kit prints that and renders the verdict unconfirmed, because the 3-day window it would be measured against is an assumption nobody supplied. The status on this line came from the ABSENCE of a row, not from its age.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, derived by running src/rulebook.py over the same values each record states; evals/check_labels.py re-runs the rule over all 308 line rows and asserts every hard case has a floor before any run may spend. The ONE label that is not derived is downstream_exception, which is planted per bucket in exact counts and appears in no corpus file.

How tight can the band be? No tolerance band in the SCORER. Exact match after trimming whitespace and punctuation for cells; integer comparison for the price gap and the slip; a closed set of seven values for the status. The two tolerances that exist -- 1 minor unit and 2 days -- are in the RULEBOOK, where they are part of the question rather than part of the marking.

Cadence: Re-score on any change to data/rulebook.json, data/fields.json, src/prompt.py or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineUnacknowledged lines, the population an incoming-document matcher cannot see

Of the order lines the supplier's acknowledgment carries no row for at all, how many reached the follow-up list -- and how many were additionally given the right one of the four exception statuses rather than some other one? A check that walks the acknowledgments it RECEIVED has no row to walk for any of them.

MethodHow the test was run

evals/judge.py, in-process, no key and no model. The same function evals/run.py calls after every run, and the same logic the free floor is scored by -- a floor and a real run scored by two different scorers is not a floor, it is a second kit.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

Read what the FLOOR does with this row rather than what the model does. The free incoming-document matcher emits NO ROW for it at all -- not confirmed, not an exception, nothing -- so the judge records it as unanswered and it reaches nobody's list. That is not a bug in the floor; it is what an acknowledgment matcher IS. This row and the 37 like it are the entire reason this kit was built, and the model path caught 38 of 38 of them. ⚑ AND NOTE WHAT THE WINDOW DOES HERE, WHICH IS NOTHING. The order sat 3 day(s) before the acknowledgment arrived. The kit prints that and renders the verdict unconfirmed, because the 3-day window it would be measured against is an assumption nobody supplied. The status on this line came from the ABSENCE of a row, not from its age.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, derived by running src/rulebook.py over the same values each record states; evals/check_labels.py re-runs the rule over all 308 line rows and asserts every hard case has a floor before any run may spend. The ONE label that is not derived is downstream_exception, which is planted per bucket in exact counts and appears in no corpus file.

How tight can the band be? No tolerance band in the SCORER. Exact match after trimming whitespace and punctuation for cells; integer comparison for the price gap and the slip; a closed set of seven values for the status. The two tolerances that exist -- 1 minor unit and 2 days -- are in the RULEBOOK, where they are part of the question rather than part of the marking.

Cadence: Re-score on any change to data/rulebook.json, data/fields.json, src/prompt.py or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe downstream prediction -- does an acknowledgment-stage mismatch predict a receiving or three-way-match exception

Of the 140 line rows that later hit a receiving or three-way-match exception, how many did the kit put on the follow-up list -- and of everything it flagged, how much came out fine? The truth here is downstream_exception, a GOLD-ONLY fact that appears in no corpus file and is deliberately not a function of the status. ⚑ THIS IS THE ONLY GRADER IN THE KIT THAT DID NOT SATURATE.

MethodHow the test was run

evals/judge.py, in-process, no key and no model. The same function evals/run.py calls after every run, and the same logic the free floor is scored by -- a floor and a real run scored by two different scorers is not a floor, it is a second kit.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

Read what the FLOOR does with this row rather than what the model does. The free incoming-document matcher emits NO ROW for it at all -- not confirmed, not an exception, nothing -- so the judge records it as unanswered and it reaches nobody's list. That is not a bug in the floor; it is what an acknowledgment matcher IS. This row and the 37 like it are the entire reason this kit was built, and the model path caught 38 of 38 of them. ⚑ AND NOTE WHAT THE WINDOW DOES HERE, WHICH IS NOTHING. The order sat 3 day(s) before the acknowledgment arrived. The kit prints that and renders the verdict unconfirmed, because the 3-day window it would be measured against is an assumption nobody supplied. The status on this line came from the ABSENCE of a row, not from its age.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, derived by running src/rulebook.py over the same values each record states; evals/check_labels.py re-runs the rule over all 308 line rows and asserts every hard case has a floor before any run may spend. The ONE label that is not derived is downstream_exception, which is planted per bucket in exact counts and appears in no corpus file.

How tight can the band be? No tolerance band in the SCORER. Exact match after trimming whitespace and punctuation for cells; integer comparison for the price gap and the slip; a closed set of seven values for the status. The two tolerances that exist -- 1 minor unit and 2 days -- are in the RULEBOOK, where they are part of the question rather than part of the marking.

Cadence: Re-score on any change to data/rulebook.json, data/fields.json, src/prompt.py or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe order routing flag against the same rule run over gold

src/extract.py::compute() over the run's OWN computed statuses, against the same rule run over gold's: does this order need somebody to DECIDE a price or a quantity, or somebody to CHASE a supplier? A business condition, so unlike the consistency diagnostic it genuinely needs labels.

MethodHow the test was run

evals/judge.py, in-process, no key and no model. The same function evals/run.py calls after every run, and the same logic the free floor is scored by -- a floor and a real run scored by two different scorers is not a floor, it is a second kit.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

Read what the FLOOR does with this row rather than what the model does. The free incoming-document matcher emits NO ROW for it at all -- not confirmed, not an exception, nothing -- so the judge records it as unanswered and it reaches nobody's list. That is not a bug in the floor; it is what an acknowledgment matcher IS. This row and the 37 like it are the entire reason this kit was built, and the model path caught 38 of 38 of them. ⚑ AND NOTE WHAT THE WINDOW DOES HERE, WHICH IS NOTHING. The order sat 3 day(s) before the acknowledgment arrived. The kit prints that and renders the verdict unconfirmed, because the 3-day window it would be measured against is an assumption nobody supplied. The status on this line came from the ABSENCE of a row, not from its age.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, derived by running src/rulebook.py over the same values each record states; evals/check_labels.py re-runs the rule over all 308 line rows and asserts every hard case has a floor before any run may spend. The ONE label that is not derived is downstream_exception, which is planted per bucket in exact counts and appears in no corpus file.

How tight can the band be? No tolerance band in the SCORER. Exact match after trimming whitespace and punctuation for cells; integer comparison for the price gap and the slip; a closed set of seven values for the status. The two tolerances that exist -- 1 minor unit and 2 days -- are in the RULEBOOK, where they are part of the question rather than part of the marking.

Cadence: Re-score on any change to data/rulebook.json, data/fields.json, src/prompt.py or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it