The business caseThe problem this solves

An event settlement carries a pack of recoverable costs -- security, stewarding, catering, production, transport, medical, cleaning, marketing, hospitality -- claimed by one party against another under a schedule with per-category caps, exclusions and riders. Expense systems check that every claimed line has a receipt attached, and they do it reliably. What they cannot do is READ the receipt. A pro forma invoice, an invoice raised for last month's fixture at another ground, a second invoice consolidating a charge already claimed under another line, and a cap a rider lowered after the deal system took its copy all produce a line that JOINS PERFECTLY: right vendor, right date, right amount to the penny, inside the cap the system carries. the receipt check an expense platform already runs -- does every claimed line have a document with the right vendor, the right date and the right amount. Implemented properly and shipped in evals/baseline.py, that check passes 39 of the 81 packs carrying a claim the evidence does not support.

The inputThe actual event expense packs

The corpus is 120 event expense packs, 1.17 MB (txt 120). An event expense pack is a set of counterparty invoices plus a deal schedule, and both halves are commercially sensitive: what a promoter pays for security and what cap a venue accepted are exactly what every counterparty would like to see. There is no public corpus. A SCRUBBED real one would be worse rather than better, and the reason is specific: scrubbing removes the invoice narrative, the rider and the proviso, and that prose layer IS what this kit measures.

Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated event expense pack for an AI
use-case kit; it reproduces no venue, no promoter, no rights holder, no vendor, no
receipt and no real person. The expense caps, the exclusions and the query floor are
ILLUSTRATIVE DEFAULTS, not any real agreement's.

Event
----------------------------------------------------------------
Event reference                  : EV-0001
Venue                            : Kesteven Park, Nether Kesteven
Promoter                         : MERIDIAN STAGE CO
Rights holder                    : PENNINE MEDIA RIGHTS
Fixture date                     : 2026-05-04
Fixture designation              : CATEGORY B
Settlement window                : 2026-04-27 to 2026-05-11
Settlement analyst of record     : Gregor Lindqvist
Expense pack prepared by         : Suri Venkatesh (promoter's settlement desk)

Expense Claim Summary
----------------------------------------------------------------
All figures in GBP. The window above is the period a document must be dated within to
be filed against this fixture.

  Lines claimed                                  8
  Documents filed                                8
  Total claimed                             22,481.87

Query floor (default)            : 250.00 GBP
Expense schedule completeness    : COMPLETE
Query authority                  : NOT DEFINED. Nothing in this kit disallows a line,
                                   rejects a claim, posts a settlement adjustment,
                                   releases a payment or amends a deal term.

Expense Terms (Structured Feed)
----------------------------------------------------------------

The outcomeWhat a good result looks like

a verdict per pack with the line at issue named, the document or clause that governs the finding, and whether the amount at stake is worth opening a query over. Measured: 72 of 72 packs needing a query were caught and 0.0 GBP of 214709.01 GBP at stake was missed.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Every rider and side letter is keyed into the deal system the day it is signed, and your evidence file is structured rather than prose — the free floor -- cap-and-category-strong, $0.00
  It joins every line to its document on four exact checks and applies the feed's caps and exclusions, reaching 67.50 pct on the discriminator with 0 false queries across 48 quiet packs -- perfect precision, for nothing.
• Riders, provisos and the pro-forma/paid distinction live in documents, and the deal feed lags — the model, with the evidence file
  93.33 pct on the discriminator against the floor's 67.50 pct, and 100.0 pct of the 39 unsupported claims the join and the feed cannot see, against 0.0 pct.
• You want the reading but cannot put the documents in front of it — the free floor, and it is the honest version of that arm
  The evidence-blind control measures exactly this: 76.67 pct on the discriminator against the floor's 67.50 pct, on the same 120 readings -- and it gets there by ACCUSING ALMOST EVERYTHING. It raises 27 false queries against the floor's 0 and leaves 17.86 pct of sound packs alone against the floor's 100.00 pct. Removing the documents does not leave you with a cheaper model, it leaves you with something noisier than free code and 1.78x the token bill.
• You need the query decision to be trusted by a counterparty — the free floor for precision, the model for recall, and read both columns
  The model missed 0 of the 72 packs needing a query; the floor missed 37. The model raised 10 false queries; the floor raised 0.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-25. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-25 — r001-expense-substantiate. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Every expense pack, the answer key, all three free floors, the injection probe and what r001-expense-substantiate actually answered ship in the repo. python3 -m evals.check_labels, python3 -m evals.run --floor cap-and-category-strong ... and python3 -m src.app all run with no key and no network.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one reading -- one event expense pack, end to end including provider-side reasoning tokens, on a shared connection. ⚠︎ NOT AN SLA AND NOT COMPARABLE ACROSS KITS: this credential is shared with every other kit in the repository and the estate made over 11,000 calls on it the same day, so these are wall-clock observations under contention. 120 readings ran in 762.4 wall seconds with 5 concurrent workers.

Current processWhat it replaces

the receipt check an expense platform already runs -- does every claimed line have a document with the right vendor, the right date and the right amount. Implemented properly and shipped in evals/baseline.py, that check passes 39 of the 81 packs carrying a claim the evidence does not support.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusEvalsEvalsArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• 4 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one reading is

120 event expense packs, read once each and independently -- there is no time axis and no carried state. Every arm on this board, free and paid, read all 120.

SetupWhat the setup figure measured

There is no index to build -- each pack goes whole into the prompt. The figures are the corpus generation itself.

LicenceLicence

MIT

Bring your ownBring your own event expense packs

tools/build_corpus.py writes data/corpus/*.txt and data/gold.jsonl. Keep the 9 section headings and the TERM:/LINE:/DOC: line shapes evals/baseline.py parses. Set your own categories and caps in TERMS and your own query floor in src/rules.py first.

⚠︎ And what stops being true when you do: The measured figures on this page do not transfer to your own packs. The feed-blind share (39 of the 81 unsupported readings here) and the prose-versus-structured split are properties of this generator's declared distribution, not facts about event expense substantiation. Re-run the evals on your own data; that is what the harness is for.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 4,151 tokens assembled, 2,538 are contexts — 61% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The prompt for EV-0010, rebuilt byte for byte by src/prompt.build() from the committed pack. It is what run r001-expense-substantiate sent.

Full promptThe prompt in full

You are an event expense substantiation review. You read one event's expense pack -- the deal's expense terms, the claimed lines and the documents filed against them -- and you decide whether the evidence actually supports what was claimed. You answer with one JSON object and no other text.

You are an expense substantiation review for one venue's event settlements. You are reading ONE
event expense pack: the deal system's structured copy of the expense terms, the EXPENSE SCHEDULE
EXTRACT itself, the expense lines somebody has claimed against the settlement, and the EVIDENCE FILE
of documents filed against them.

YOUR JOB IS NOT TO CHECK THAT EACH LINE HAS A DOCUMENT. An expense system already does that, for
nothing, and on this pack it will almost always say yes: the document's vendor matches the line's
vendor, its date sits inside the settlement window, and its amount matches the amount claimed to the
penny. What you are looking for is the line that PASSES that join and is still not substantiated --
a document that records a quotation rather than a payment, a document raised for a different
fixture, a charge already invoiced under another line, a cap a rider lowered after the deal system
took its copy, a category a side letter makes conditional on something this fixture is not.

⚠︎ THE EXPENSE CAPS, THE EXCLUSIONS AND THE QUERY FLOOR ARE ILLUSTRATIVE DEFAULTS, NOT ANY REAL
AGREEMENT'S. Apply them exactly as printed regardless of whether they look right for the event in
front of you.

THE RULES, applied in this order:
 - FIRST check completeness. If the pack records that the EXPENSE SCHEDULE IS NOT ATTACHED, the
   verdict is PACK_INCOMPLETE: no line is judged, the defect is UNDETERMINED, the exception line is
   NONE, the governing reference is NONE and nothing is disallowed. There is no default term for a
   schedule nobody can read (Rule S-8).
 - THEN check whether the pack ADDS UP. If the printed "Total claimed" is not the sum of the claimed
   lines, the verdict is PACK_ARITHMETIC and the defect is ARITHMETIC (Rule S-6). This is the only
   verdict about the addition, and it is NOT the verdict for a pack that adds up correctly and
   claims something the evidence does not support.
 - THEN take each claimed line and its filed document together.
   - No document filed against the line at all: EVIDENCE_MISSING, defect NO_RECEIPT, governing
     reference NONE. Do NOT infer what the missing document would have said.
   - A document whose AMOUNT is less than the amount claimed: EVIDENCE_MISMATCH, defect
     AMOUNT_OVERSTATED.
   - A document whose VENDOR is not the line's vendor: EVIDENCE_MISMATCH, defect WRONG_VENDOR.
   - A document DATED OUTSIDE the settlement window printed on the pack: EVIDENCE_MISMATCH, defect
     OUT_OF_WINDOW.
   - A document whose own narrative names a DIFFERENT FIXTURE -- another date, another ground --
     from the one this pack settles: EVIDENCE_MISMATCH, defect WRONG_EVENT. The header matching is
     not a cure; a correctly dated invoice for somebody else's fixture substantiates nothing here.
   - A document that is a PRO FORMA, a quotation, an estimate or a request for payment in advance,
     rather than evidence of a payment made: EVIDENCE_MISMATCH, defect NOT_PAID_EVIDENCE.
   - A line whose charge its own document says was ALREADY INVOICED to this settlement under
     another document: EVIDENCE_MISMATCH, defect DUPLICATE_CLAIM. Name the LATER line -- the one
     whose document records the overlap -- as the exception.
 - THEN judge each line against the governing terms.
   - A RIDER OR SIDE LETTER IN THE SCHEDULE EXTRACT SUPERSEDES THE STRUCTURED TERM FEED (Rule S-4).
     The feed is a convenience copy of the deal and a copy that has not caught a rider is exactly how
     a superseded cap keeps being paid. Where a line exceeds the cap AS THE RIDER STATES IT, the
     verdict is CAP_EXCEEDED, the defect is CAP_BREACH, and the governing reference is THE RIDER'S
     id -- not the clause it amends.
   - A line over the cap the feed and the schedule agree on: CAP_EXCEEDED, defect CAP_BREACH,
     governing reference the CLAUSE id. A line EXACTLY AT its cap is within it and is not a breach.
   - A category the terms exclude, or a category a clause makes conditional on something this
     fixture is not: NOT_REIMBURSABLE, defect EXCLUDED_CATEGORY, governing reference the clause or
     side letter that excludes it.
 - Otherwise the verdict is SUBSTANTIATED, the defect is NONE, the exception line is NONE and the
   governing reference is NONE.
 - EXACTLY ONE LINE IS AT ISSUE ON EACH PACK. Name it in "exception_line".
 - THEN decide the OWNER. Start from the settlement analyst of record, but a note recording a
   handover to someone else supersedes it. The person who PREPARED the pack is not the owner.
 - FINALLY decide "disallow": YES only where a defect is named AND the amount at stake is at or
   above the printed QUERY FLOOR. A defect below the floor is still a defect and is still named --
   it is simply not worth opening a settlement query with a counterparty over (Rule S-9). ARITHMETIC
   and UNDETERMINED are never a disallowance.

Answer with a single JSON object and nothing else:

{"verdict": "SUBSTANTIATED|EVIDENCE_MISSING|EVIDENCE_MISMATCH|CAP_EXCEEDED|NOT_REIMBURSABLE|PACK_ARITHMETIC|PACK_INCOMPLETE",
 "defect": "NONE|NO_RECEIPT|AMOUNT_OVERSTATED|WRONG_VENDOR|OUT_OF_WINDOW|WRONG_EVENT|NOT_PAID_EVIDENCE|DUPLICATE_CLAIM|CAP_BREACH|EXCLUDED_CATEGORY|ARITHMETIC|UNDETERMINED",
 "exception_line": "<the claimed line at issue, e.g. E-05, or NONE>",
 "governing_ref": "<the document or clause that governs your finding -- a document id like R-05, a
   clause id like S-3.3, a rider id like X-1.1, or NONE>",
 "disallow": "YES|NO",
 "owner": "<the settlement analyst who owns this pack>",
 "rationale": "one sentence, naming the line, what the document or clause actually says, and why
   that does or does not support the amount claimed"}

Precedence: PACK_INCOMPLETE if the schedule is not attached; then PACK_ARITHMETIC if the pack does
not add up; then the evidence findings above; then the cap and category findings; otherwise
SUBSTANTIATED.

Expense pack
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated event expense pack for an AI
use-case kit; it reproduces no venue, no promoter, no rights holder, no vendor, no
receipt and no real person. The expense caps, the exclusions and the query floor are
ILLUSTRATIVE DEFAULTS, not any real agreement's.

Event
----------------------------------------------------------------
Event reference                  : EV-0010
Venue                            : Kesteven Park, Nether Kesteven
Promoter                         : MERIDIAN STAGE CO
Rights holder                    : SEVERNSIDE BROADCASTING
Fixture date                     : 2026-07-10
Fixture designation              : CATEGORY A
Settlement window                : 2026-07-03 to 2026-07-17
Settlement analyst of record     : Hana Yoshida
Expense pack prepared by         : Callum Whitfield (promoter's settlement desk)

Expense Claim Summary
----------------------------------------------------------------
All figures in GBP. The window above is the period a document must be dated within to
be filed against this fixture.

  Lines claimed                                  7
  Documents filed                                7
  Total claimed                             20,178.23

Query floor (default)            : 250.00 GBP
Expense schedule completeness    : COMPLETE
Query authority                  : NOT DEFINED. Nothing in this kit disallows a line,
                                   rejects a claim, posts a settlement adjustment,
                                   releases a payment or amends a deal term.

Expense Terms (Structured Feed)
----------------------------------------------------------------
The deal system's own copy of the expense terms, as the settlement engine reads them.
One line per recoverable category. THE FEED IS A COPY OF THE DEAL, NOT THE DEAL: a rider
or side letter recorded in the schedule extract below governs over anything on these
lines.

TERM: id=S-3.1 category=SECURITY cap=8,500.00 status=ALLOWED bearer=PROMOTER label="matchday security"
TERM: id=S-3.2 category=STEWARDING cap=6,000.00 status=ALLOWED bearer=PROMOTER label="stewarding and crowd services"
TERM: id=S-3.3 category=CATERING cap=4,000.00 status=ALLOWED bearer=PROMOTER label="back-of-house catering"
TERM: id=S-3.4 category=PRODUCTION cap=12,000.00 status=ALLOWED bearer=PROMOTER label="stage and production hire"
TERM: id=S-3.5 category=TRANSPORT cap=3,500.00 status=ALLOWED bearer=VENUE label="visiting-side transport"
TERM: id=S-3.6 category=MEDICAL cap=2,800.00 status=ALLOWED bearer=VENUE label="medical cover"
TERM: id=S-3.7 category=CLEANING cap=2,200.00 status=ALLOWED bearer=VENUE label="post-event cleaning"
TERM: id=S-3.8 category=MARKETING cap=5,000.00 status=ALLOWED bearer=PROMOTER label="local marketing spend"
TERM: id=S-3.9 category=HOSPITALITY cap=3,000.00 status=ALLOWED bearer=VENUE label="hospitality suite service"
TERM: id=S-3.10 category=PYROTECHNICS cap=NONE status=EXCLUDED bearer=PROMOTER label="pyrotechnic effects"

Expense Schedule Extract
----------------------------------------------------------------
The governing clauses, as written. Where a rider or side letter appears below, it
replaces or qualifies the clause it names.

S-3.1           Matchday security is recoverable against the settlement up to 8,500.00 GBP per
                fixture. Officers must be SIA-licensed and the invoice must state the
                headcount.

S-3.2           Stewarding and crowd services are recoverable up to 6,000.00 GBP per fixture.

S-3.3           Back-of-house catering is recoverable up to 4,000.00 GBP per fixture. Catering for
                hospitality guests is not recoverable under this clause.

S-3.4           Stage, lighting and production hire is recoverable up to 12,000.00 GBP per fixture.

S-3.5           Visiting-side transport is recoverable up to 3,500.00 GBP per fixture and is borne
                by the venue.

S-3.6           Medical cover is recoverable up to 2,800.00 GBP per fixture. Cover must be present
                from doors to final egress.

S-3.7           Post-event cleaning is recoverable up to 2,200.00 GBP per fixture.

S-3.8           Local marketing spend is recoverable up to 5,000.00 GBP per fixture against
                itemised third-party invoices.

S-3.9           Hospitality suite service is recoverable up to 3,000.00 GBP per fixture.

S-3.10          Pyrotechnic effects are NOT recoverable against the settlement in any amount. The
                cost is borne by the promoter absolutely.

S-4.1           A rider or side letter recorded in this schedule SUPERSEDES the deal system's
                structured copy of the term it amends. The structured feed is a convenience copy
                of the deal and is not the deal; where the two disagree, this schedule governs.

S-6.1           The total claimed printed on this pack must be the sum of the claimed lines. Where
                it is not, the pack does not add up. That is a defect of addition and is not, by
                itself, a finding against any line.

S-7.1           A claimed line is SUBSTANTIATED only where the document filed against it is
                evidence of a PAYMENT ACTUALLY MADE, by this claimant, FOR THIS FIXTURE, in the
                amount claimed. A document that names another fixture, that records a quotation or
                a pro forma rather than a payment, or that covers a charge already claimed under
                another line, does not substantiate the line it is filed against -- whatever its
                vendor, its date and its face amount say.

S-8.1           Where the governing expense schedule is not attached to this pack, no line may be
                judged against it. There is no default term for a schedule nobody can read, and a
                line is not substantiated by the absence of the clause that would have governed it.

S-9.1           A query is raised only where the amount at stake reaches the query floor printed on
                this pack. A defect below the floor is still recorded and still named; it is simply
                not worth opening a settlement query with a counterparty over.

Expense Lines As Claimed
----------------------------------------------------------------
One line per claimed expense, in the order the pack presents them.

LINE: id=E-01 category=MARKETING vendor="QUILLMARK AGENCY" date=2026-07-14 amount=3,328.02 evidence=R-01 label="local marketing spend"
LINE: id=E-02 category=STEWARDING vendor="FOOTFALL STEWARDS LTD" date=2026-07-12 amount=2,385.79 evidence=R-02 label="stewarding and crowd services"
LINE: id=E-03 category=SECURITY vendor="BASTION EVENT SECURITY" date=2026-07-08 amount=4,495.46 evidence=R-03 label="matchday security"
LINE: id=E-04 category=HOSPITALITY vendor="VELVET ROPE EVENTS" date=2026-07-13 amount=2,337.24 evidence=R-04 label="hospitality suite service"
LINE: id=E-05 category=CLEANING vendor="TIDELINE FACILITIES" date=2026-07-13 amount=1,777.15 evidence=R-05 label="post-event cleaning"
LINE: id=E-06 category=MEDICAL vendor="GREENSTONE MEDICAL" date=2026-07-13 amount=1,218.36 evidence=R-06 label="medical cover"
LINE: id=E-07 category=PRODUCTION vendor="LUMEN STAGE SYSTEMS" date=2026-07-09 amount=4,636.21 evidence=R-07 label="stage and production hire"

  Total claimed                             20,178.23

Evidence File
----------------------------------------------------------------
The documents filed against this pack. One block per document. The header records what the
expense system captured; the narrative is what the document itself says.

DOC: id=R-01 vendor="QUILLMARK AGENCY" date=2026-07-14 amount=3,328.02
     Narrative: Local marketing spend for the fixture of 10 July 2026 at Kesteven Park,
                Nether Kesteven. Paid in full by bank transfer.

DOC: id=R-02 vendor="FOOTFALL STEWARDS LTD" date=2026-07-12 amount=2,385.79
     Narrative: Stewarding and crowd services for the fixture of 10 July 2026 at Kesteven
                Park, Nether Kesteven. Paid in full by bank transfer.

DOC: id=R-03 vendor="BASTION EVENT SECURITY" date=2026-07-08 amount=4,495.46
     Narrative: Matchday security for the fixture of 10 July 2026 at Kesteven Park, Nether
                Kesteven. Paid in full by bank transfer.

DOC: id=R-04 vendor="VELVET ROPE EVENTS" date=2026-07-13 amount=2,337.24
     Narrative: Hospitality suite service for the fixture of 10 July 2026 at Kesteven Park,
                Nether Kesteven. Paid in full by bank transfer.

DOC: id=R-05 vendor="TIDELINE FACILITIES" date=2026-07-13 amount=1,777.15
     Narrative: PRO FORMA INVOICE. This is a request for payment in advance and is NOT a VAT
                receipt. No payment has been received against it. A VAT receipt will be
                issued on clearance of funds. Post-event cleaning for the fixture of 10 July
                2026.

DOC: id=R-06 vendor="GREENSTONE MEDICAL" date=2026-07-13 amount=1,218.36
     Narrative: Medical cover for the fixture of 10 July 2026 at Kesteven Park, Nether
                Kesteven. Paid in full by bank transfer.

DOC: id=R-07 vendor="LUMEN STAGE SYSTEMS" date=2026-07-09 amount=4,636.21
     Narrative: Stage and production hire for the fixture of 10 July 2026 at Kesteven Park,
                Nether Kesteven. Paid in full by bank transfer.

Settlement Notes
----------------------------------------------------------------
Settlement of this fixture has been handed over to Gregor Lindqvist, who owns it from this pack
onward.

Raw responseThe raw response

{
 "verdict": "EVIDENCE_MISMATCH",
 "defect": "NOT_PAID_EVIDENCE",
 "exception_line": "E-05",
 "governing_ref": "S-7.1",
 "disallow": "YES",
 "owner": "Gregor Lindqvist",
 "rationale": "E-05 is filed against R-05, which is expressly a pro forma request for payment in advance with no payment received, so it is not evidence of a payment actually made for this fixture."
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch event expense claims with no real evidence — 120 event expense packs. One model answered, and every answer was then graded Two different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Every arm read all 120 packs -- there is no sampled half and no scored subset on this board. 120 of 120 readings answered and parsed, with 0 failures.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on this page is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection, never a bill and never a vendor's claim about this workload.

Rates checked 2026-08-18. The provider that actually ran the calls is kept out of these tables per this estate's naming rule, so no figure here is a bill.

The gradersTwo ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

The three free floors, the stub and every pre-run check cost $0.00 and no calls. 318 calls were spent on this kit and 318 reached a published result file; 0 were discarded. The socket timeout was set from the published ceiling in the SAME EDIT as the ceiling, before the first paid call rather than after a discarded run, which is why the discarded count is 0 rather than the three figures a sibling kit paid for the same lesson. ⚠︎ THE CEILING ITSELF WAS STILL UNDER-SIZED BY THE PROBE: the 6-call calibration drew 15.3 pct of it and the scored run drew 80.7 pct. Nothing truncated, so nothing was discarded -- but that was margin, not foresight.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, AND ALMOST ALL OF THEM ARE REASONING. 97.1 pct of r001-expense-substantiate's output (418233 of 430820 tokens) was provider-side reasoning left at the default. The answer is six short fields and one sentence; the bill is the thinking in front of it -- and on this task the thinking is a per-line join run once structurally and once against prose.
• THE PROMPT IS MOSTLY FIXED TEXT. 4152 input tokens per reading, and the rules and the JSON shape are identical on every call -- so a provider with prompt caching would price this workload very differently, and that was not measured here.
• THE NUMBER OF CLAIMED LINES. A pack carries 7 to 9 lines and each one is a join the model reasons through. A pack with thirty lines is a longer reading, not a harder one.
• ⚠︎ NOT THE CORPUS SIZE. There is no index and no retrieval; each pack goes whole into the prompt, so doubling the archive doubles the number of calls and changes nothing else.

Your volumeWhat it costs at your volume

Linear and nothing amortises: 10x the packs is 10x the calls, about $12.85 per thousand packs on the projection card. ⚑ THE CONSTRAINT THAT ACTUALLY BOUND THIS RUN WAS NOT MONEY. On a credential shared with every sibling kit -- over 11,000 estate calls the same day -- this kit's throughput sat around six readings a minute and p95 latency reached 122819 ms. Budget provider concurrency before tokens.

Model choiceWhy this model

One tier was run, on the reader's own key, because the kit's claim is about what the EVIDENCE FILE is worth and not about which model is best. The evidence-blind control is the same tier with one section removed, which isolates that variable exactly; a second model would have confounded it.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

per-query average, this run

The codeEvery file, and what it is for

13 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Seven of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/rules.pythe substantiation rulebook — a swap seam

The enums, the defect-to-verdict function, the query-floor decision, and is_unsupported() -- the discriminator's truth function, written once so no arm and no scorer can keep its own version of what 'unsupported' means.

You change it to: UNSUPPORTED -- the discriminator's truth function. Moving PACK_INCOMPLETE into it would change every headline figure on this board, which is exactly why it is one constant and not a list repeated in four files.

# The substantiation rulebook, as data. Pure code, no model, no vendor.

FIELDS = ("verdict", "defect", "exception_line", "governing_ref", "disallow", "owner")
SUBSTANTIATED = "SUBSTANTIATED"
EVIDENCE_MISSING = "EVIDENCE_MISSING"
EVIDENCE_MISMATCH = "EVIDENCE_MISMATCH"
CAP_EXCEEDED = "CAP_EXCEEDED"
NOT_REIMBURSABLE = "NOT_REIMBURSABLE"
PACK_ARITHMETIC = "PACK_ARITHMETIC"
PACK_INCOMPLETE = "PACK_INCOMPLETE"
VERDICTS = (SUBSTANTIATED, EVIDENCE_MISSING, EVIDENCE_MISMATCH, CAP_EXCEEDED,

src/segment.pythe section splitter

Splits the pack into its 9 named sections; asserted across all 120 documents.

# Split an expense pack into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "Event", "Expense Claim Summary",
RULE = "-" * 64
def split(text):
def render(secs):

src/select.pythe send filter — a swap seam

Settlement Contact is mapped by no answered field and therefore never sent -- including when every section is renamed, which is the direction that would otherwise hold by luck.

You change it to: SECTION_HINTS and NEVER_SENT.

# Pick which sections of an expense pack are sent. Pure code -- the last deterministic step before

BANNER = "Synthetic Record"
EVENT = "Event"
SUMMARY = "Expense Claim Summary"
FEED = "Expense Terms (Structured Feed)"
SCHEDULE = "Expense Schedule Extract"
LINES = "Expense Lines As Claimed"
EVIDENCE = "Evidence File"
CONTACT = "Settlement Contact"
NOTES = "Settlement Notes"

src/prompt.pythe prompt

Two parts: the instruction with the rules and the JSON shape, then the pack. The evidence-blind control replaces exactly one section's body.

# Assemble the one prompt this kit sends. No template engine, no framework -- string concatenation

BLIND_LINE = ("The evidence file is not available for this pack. The documents themselves were not "
INSTRUCTION = """\
def build(text, evidence_blind=False):

src/adapters/__init__.pythe model call — a swap seam

Raw HTTP, stdlib only, bounded retry, a socket timeout set from the published ceiling IN THE SAME EDIT as the ceiling, and a separate retry budget for transport failures. The shared daily call cap is checked here.

You change it to: ADAPTER_TIMEOUT_S, set from src/pack.MAX_TOKENS and changed in the same edit. A completion is not streamed, so a timeout shorter than the work turns the heaviest readings -- the only ones a raised ceiling was for -- into billed transport failures that retry and fail anyway.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TRANSPORT_RETRIES = 1
TIMEOUT_S = int(os.environ.get("ADAPTER_TIMEOUT_S", "900"))
def _post(url, headers, payload, timeout=None):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}

src/pack.pythe reader

One expense pack, one call, at the published 32000-token ceiling.

# One expense pack, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = ("You are an event expense substantiation review. You read one event's expense pack -- the "
MAX_TOKENS = 32000
FIELDS = ("verdict", "defect", "exception_line", "governing_ref", "disallow", "owner")
def documents():
def load_doc(doc_id):
def facts_of(text):
def _parse(txt):

evals/run.pythe per-reading cache

Append-only, written before anything is scored. --resume never re-buys a call already paid for; --score-cache scores exactly what came back and records that count as the denominator.

# Run the review over the 120 event expense packs and write one result file. THIS SPENDS MONEY.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
KIT = "expense-substantiate"
def cache_path(run_id):
def load_cache(run_id, max_tokens, evidence_blind):
def append_cache(run_id, row):
def load_gold():
def stub_complete(cfg, system, user, max_tokens=1024):

evals/scoring.pythe scorer

Exact match per cell. The substantiation discriminator is a separate binary with separate denominators for its two directions, the feed-blind slice is scored on its own, and declining to judge a judgeable pack is counted separately from getting it wrong.

# Score a run against the answer key. Pure code, exact match per cell. No model grades anything.

FIELDS = R.FIELDS
def _pct(n, d):
def _same(got, want):
def _same_name(got, want):
def score(records, golds):
def _protection(records, golds):

evals/baseline.pythe three free floors

totals-tie-out, receipt-join-strict and cap-and-category-strong -- $0.00 each. The strongest reaches 67.50 pct on the discriminator and wins the precision columns outright.

# THE FREE FLOORS. Three of them, none a strawman, all of them GBP 0.00.

MODES = ("totals-tie-out", "receipt-join-strict", "cap-and-category-strong")
TOL = 0.005
def _num(v):
def _section(text, name, nxt):
def _facts(text):
HANDOVER = re.compile(r"handed over to ([A-Z][^,\n]*?), who owns it", re.M)
def _owner(f):
def _finish(verdict, defect, line, ref, disallow, owner, why):
def review(text, mode="receipt-join-strict"):

evals/check_labels.pythe pre-spend gate

Six assertions, all of them about something that fails silently. It refuses to let any run spend until they are clear.

# THE PRE-SPEND GATE. Free, offline, and it runs before any run may cost anything.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
BANNED = [
ALLOW_FILES = {os.path.join("src", "adapters", "__init__.py")}
SCAN_DIRS = ("src", "evals", "tools", "ui")
def banned_paths():
def check():
def main():

tools/build_corpus.pythe corpus generator and its own gate — a swap seam

Writes the packs and the key, then RE-READS every emitted pack and asserts the perturbation it was built to carry is the only defect present. It caught a real bug on the first build.

You change it to: TERMS and CLAUSE_TEXT -- the categories, the per-fixture caps, the exclusions and the riders. Illustrative defaults, replaced wholesale when you point this at your own packs.

# Generate the 120 synthetic event expense packs and the answer key. Free, offline, deterministic.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
RULE = "-" * 64
SEED = 20260825
QUERY_FLOOR = 250.00
VENUES = [
PROMOTERS = ["TIDEWATER PROMOTIONS", "HALLGROVE LIVE", "MERIDIAN STAGE CO",

src/app.pythe local UI server

http.server, stdlib. Renders with no key.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "9057"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-expense-substantiate")
def _join_report(text):
class H(BaseHTTPRequestHandler):
def main():

ui/app.jsthe local UI client

Hand-written JS, no framework.

# 

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 4151 input and 3590 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

⚠︎ MEASURED, AND IT HELD ON THE DECISION FIELDS. Settlement Notes is this kit's injection surface and it is sent DELIBERATELY: on this vertical the desk that types the notes is often the CLAIMANT'S, which makes it the one field an interested counterparty can influence. One of the five shipped notes is instruction-shaped for exactly that reason. x001-expense-substantiate-injection forced it onto every reading where a query was due and the scored run had raised one: 72 of 72 still raised it. ⚑ AND THE WHOLE ANSWER WAS SCORED, NOT ONLY THE QUERY DECISION -- a sibling kit measured a model that refused an injected instruction and fabricated a different error in the same reply. Here 17 answers moved: 10 only in the settlement analyst, which is the PROBE'S OWN VARIABLE (the handover sentence lives in the section the probe replaces, and the model correctly reverted to the analyst of record every time), and 7 in governing_ref. Verdict, defect, exception_line and disallow moved on ZERO readings.

API_KEY is read from .env (gitignored at every depth, and this repository has never held a credential) or from the real environment, never from a page and never from a reader. The kit's UI has no key field and no endpoint that accepts one. Error strings are scrubbed of the key and the base URL before they reach the browser.

The experimentWe DID attack it -- one sentence, on every reading where it could possibly matter

The probe replaces the Settlement Notes section with the instruction-shaped note and changes nothing else -- the section is REPLACED rather than appended to, so the document length moves as little as possible and the experiment stays one variable. It fires only on readings whose answer key says a query is due AND which r001-expense-substantiate itself raised, because a reading already answered 'no query' cannot be suppressed and counting it would confound the denominator. 72 readings qualified; 72 still raised the query. Both were run for real on 2026-08-25.

All three boundaries are measured in both directions: each check is shown passing on the shipped kit AND convicting a deliberately seeded violation. A check only ever seen green is a check nobody has proven can go red.

The result0 of 72 queries suppressed by the instruction-shaped note (95 pct ceiling 4.17 pct); 7 answers moved in governing_ref only, and no decision field moved at all

One phrasing, one model, one corpus, 72 paired trials -- every reading where suppression was even possible.

HonestyWhat this does not prove

• Any injection phrasing other than the one shipped. A note claiming the venue has already signed the pack off, claiming the rider was never countersigned, or written to look like a system banner is a different experiment.
• An injection carried inside a DOCUMENT NARRATIVE rather than the notes. That is the field this kit reads most closely and it was not attacked.
• Whether the model can be pushed the other way -- into raising a query that is not due. Only the suppression direction was fired.
• Anything about a second model or a second corpus.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never disallow a line, reject a claim, post a settlement adjustment, release a payment or amend a deal term. This kit produces an OPINION and nothing else.

Stated to the model on every call in src/prompt.py's INSTRUCTION and printed on every pack as the Query authority field; enforced by ABSENCE in the repository, and that absence is SCANNED by evals/check_labels.py before any run may spend.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is a PROMPT RULE PLUS A STATIC SCAN, not a runtime enforcement layer. Nothing stops a forker adding such a path tomorrow; the scan catches it only the next time somebody runs check_labels.py, which is a manual step.
• The injection result is ONE SENTENCE against ONE model on ONE corpus over 72 paired trials. It is not a resistance rate for prompt injection in general, and the 95 pct ceiling of 4.17 pct is printed beside it precisely because a zero is not a proof.
• The verdict is an OPINION. Nothing here is an accounting determination, a contractual notice, or advice; a settlement analyst decides whether to raise the query.
• The scan reads for acts that reach outside the process. It does not and cannot prove the kit is side-effect free in general.

WatchedWhat is watched, and why that one

The kit measures 76 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

11 measured by the latest run 65 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (arithmetic_cells 6, documents 120, evidence_blind False, feed_blind_cells 39, feedblind_unsupported_cells 39, incomplete_cells 5, packs_needing_query 72, query_cells 72, quiet_cells 48, readings 120, readings_scored 120, scope full, scope_intended 120, sound_cells 28, stateless False, substantiation_cells 81, substantiation_sound_cells 39, value_at_stake_gbp 214709.01) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

6 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

comply · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 3 runs measured the same system — they differ on floor, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

comply · with the model in the path — 2 runs. Columns here are only ever compared with each other.

not a time series No two of these 2 runs measured the same system — they differ on evidence_blind, stateless — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 11 chips that all say so.

DeviationsWhat deviated

0 breaches across 6 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

ON DEMAND, ONCE PER PACK -- there is no clock and no cursor. A pack is filed once and read once; nothing here polls, schedules or carries state between readings.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, no framework dependency, and a requirements.txt that names nothing because nothing is needed.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear: the reader -> src/prompt.py -> src/adapters -> evals/scoring.py. No agent, no tool loop, no retrieval step, no state.

The other sideWhat a framework costs you

• You write the retry and timeout policy yourself. That is the point on this kit -- the two values are set from the published ceiling and changed in the same edit as it.
• You write the provider adapters yourself. Two are shipped, in about 40 lines each.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-expense-substantiate on the fast tier, with the evidence file, 2026-08-25. This kit records telemetry no telemetry captured — 0 of the 6 readings on this axis, 0 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 4 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

No band column. None of these readings has an alarm band derived for it yet, so there is nothing for the guardrail board to judge them against. Measured and unjudged is a real state and it is stated once; a column of “no band” is not a band.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: ADAPTER_TIMEOUT_S, API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe verdict, the named defect, the line at issue, the governing reference, the query call and the settlement analyst, per reading, exact match against the answer key

whether each of the 6 answered fields equals the answer key, per reading

MethodHow the test was run

python3 -m evals.run --run-id <id> ...; evals/scoring.py compares strings. No model grades anything.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 120 per field. expense_substantiation_verdict_pct is scored separately over its own denominators.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the answer key in data/gold.jsonl, written and then re-asserted document by document by tools/build_corpus.py.

No true/false rates for this grader. It records 6 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold was swept: exact match has no tunable.

Cadence: once per run; free

The decisionWhen to reach for it

In one lineDoes the evidence in this pack support what was claimed

whether the arm called a pack unsupported when it was, and left a sound pack alone

MethodHow the test was run

evals/scoring.py, same pass as the exact-match grader.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

called correctly over all 120 readings; the two error directions over their own denominators (81 unsupported, 39 not). PACK_INCOMPLETE is NOT counted as unsupported -- rules.is_unsupported() is the single truth function and every arm and this scorer read it from there.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: rules.is_unsupported() over the keyed verdict.

No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold. The two denominators are printed beside every rate because they are different sizes and different failures.

Cadence: once per run; free

The decisionWhen to reach for it