The business caseThe problem this solves

Every extraction benchmark runs on clean text. Production runs on scans and on documents where two suppliers label the same field three different ways. Nobody publishes what that costs you. A hand-check of every scanned document, or the unexamined assumption that the extractor which scored well in the demo will hold up on the post room's output.

The inputThe actual business documents

The corpus is 120 business documents, 0.08 MB (txt 120). Measuring this gap needs the SAME page in two conditions with one ground truth attached to both, and no public corpus ships that pair. Real scanned enterprise documents cannot be published at all — third-party, usually confidential, licences that do not permit redistribution. Self-authoring removes the argument and is the only way the degraded half can be held to a known answer: we know what the page said before it was damaged, because we wrote it.

IRONBRIDGE FREIGHT
Trade Counter  |  Bristol

INVOICE

Reference Number: INV-69527
Date: 2026-01-12
Remit To: Ironbridge Freight

Description                         Qty      Line total
Packing tape, clear                    3         817.98
Label rolls, 100x50                    2         932.47
Label rolls, 100x50                   33         876.25

Subtotal                                        2626.70
Tax                                            37844.41
Grand Total (GBP)                              40471.11

Payment terms: 60 days from date of issue.
Registered in England. VAT 753754702.

The outcomeWhat a good result looks like

One number per method: how many accuracy points you lose when the input stops being clean. The model loses 11.8. Label-anchored rules lose 26.4.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Your documents arrive clean and digitally generated — The free rules baseline
  Rules score 88.2% on clean input for $0.00. The model's 100.0% is better, but that is an 11.8-point gap you are paying for on text that was never damaged.
• Your documents arrive scanned, and identifiers matter — The model, plus a check digit or cross-reference on the identifiers
  The model holds 88.2% on scanned input against the floor's 61.8%, but its document-number field is the weakest at 73.3% — the field you most need to trust degrades most.
• You cannot tolerate an invented value — Either — both scored zero inventions
  Neither method invented a value in either condition. ⚠︎ That rests on 20 not-stated cells, which is the thinnest denominator on this page.

At a glanceHow the whole thing runs

Run twice over the same set, for real, the last on 2026-08-14. Every figure on these pages was captured from those runs — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-14 — r001-clean + r002-messy. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — git clone, then python3 tools/build_corpus.py && python3 evals/baseline.py clean && python3 evals/baseline.py messy. No dependencies, no key, ~3 seconds, and you have the free floor's half of the result.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per document, scanned condition

Current processWhat it replaces

A hand-check of every scanned document, or the unexamined assumption that the extractor which scored well in the demo will hold up on the post room's output.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• clean
• scanned
• 2 of 9 stations carry a recorded failure

scroll → 9 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one document is

No split — nothing is fitted. Every document is scored in both conditions.

size_p50/p95 are 0 because there is no segmentation: each document is passed whole. Recorded as an explicit zero rather than omitted, so the absence is a decision and not a missing key.

SetupWhat the setup figure measured

There is no index. The kit does not retrieve — each document is passed whole to one call. Corpus preparation is generation, not indexing: tools/build_corpus.py writes 120 files from a fixed seed in under a second, with no model and no key.

LicenceLicence

MIT. Granted by us because we wrote every byte of it — the generator is in the kit and its output is a pure function of seed 20260814. Nothing is derived from a third-party corpus, so there is no upstream licence to honour and no attribution owed. Verified 2026-08-14 by rebuilding from a clean tree and diffing byte for byte (tools/build_corpus.py --verify).

Bring your ownBring your own business documents

Drop your own text files into data/corpus/clean/ and data/corpus/messy/ with matching names, add their answers to data/gold.jsonl, and both methods run unchanged. If you have real scanned documents with ground truth, that is the single most valuable thing you can contribute here — it replaces the simulation this kit is honest about being.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 399 tokens assembled, 257 are instructions — 64% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured by prefix subtraction against the provider's own tokenizer (evals/prompt_tokens.py, 2 calls at max_tokens=1) on document d000, not estimated from characters.

Full promptThe prompt in full

Read the document below and return this JSON object:

{
  "doc_number": null,   // Document number The document's own identifier, not any other reference on the page.,
  "doc_date": null,   // Document date ISO format, YYYY-MM-DD.,
  "total_amount": null,   // Total amount due The final total, not the subtotal and not a line total. Digits only, two decimals.,
  "currency": null,   // Currency Three-letter ISO code.,
  "counterparty": null,   // Counterparty The organisation that issued the document.,
  "reference": null,   // Purchase-order or customer reference Often absent. If the page does not state one, say so rather than guessing.
}

Rules:
- Copy values as the document states them. Do not reformat except where a field says to.
- If the document does not state a field, return null for it. Do not guess, and do not
  substitute a similar-looking value from elsewhere on the page.
- Return the JSON object and nothing else.

DOCUMENT
--------
IRONBRIDGE FREIGHT
Trade Counter  |  Bristol

INVOICE

Reference Number: INV-69527
Date: 2026-01-12
Remit To: Ironbridge Freight

Description                         Qty      Line total
Packing tape, clear                    3         817.98
Label rolls, 100x50                    2         932.47
Label rolls, 100x50                   33         876.25

Subtotal                                        2626.70
Tax                                            37844.41
Grand Total (GBP)                              40471.11

Payment terms: 60 days from date of issue.
Registered in England. VAT 753754702.

Raw responseThe raw response

{
  "doc_number": "INV-69527",
  "doc_date": "2026-01-12",
  "total_amount": "4047.11",
  "currency": "GBP",
  "counterparty": "ironbridge freight",
  "reference": null
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Read scanned supplier invoices and know which fields to trust — 720 field cells drawn from 120 real business documents. One model answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

Exact match after a normalisation applied identically to gold and prediction (whitespace, trailing punctuation, case; money to two decimals; codes uppercased and de-spaced). The same scorer runs both methods — a comparison whose two sides are scored by two pieces of code is not a comparison.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Run it twiceThe same set, run again

—

Grading costWhat it costs

Every dollar here is a MEASURED token count multiplied by a named vendor's PUBLISHED rate, read from build/facts/models.json.

Rates checked 2026-07-17.

Grading unitWhat the grading figure prices

The GRADER is free: exact match after a shared normalisation, pure code, no model in the scoring loop. Re-scoring the committed run records costs $0.00 and a fresh clone can do it with no key at all. The model calls that produced the answers are the system under test, not the evaluation of it — those are priced per document in the Cost lens.

The gradersOne way to grade, and why it is the only one

It scores 88.2% clean and 61.8% messy. The model reading a SCAN (88.2%) is as accurate as the rules reading a PERFECT page (88.2%).

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-07-17. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingGrading is free here, and it is a measured zero

ZERO, AND A MEASURED ZERO RATHER THAN AN UNPRICED ONE. evals/score.py makes no call and needs no key: grading all 720 cells costs a forker nothing, and the free floor that supplies half the comparison costs nothing either. A fresh clone reproduces the rules half of this page — both conditions — with no API key at all.

Cost driversWhat actually moves the bill

• Input is the INSTRUCTION, not the document — 257 of 399 tokens on a typical page. That is the opposite of the retrieval kits, where context dominates: cut the field hints and you cut most of the input bill.
• One call per document, so cost scales linearly with corpus size. No batching, no caching.
• Output is small and stable at ~63 tokens because the reply is a six-field JSON record — but only with the provider's reasoning pass disabled. With it on the same reply cost 279.
• A scanned page costs marginally MORE than a clean one: the damage adds tokens, it does not remove them.

Your volumeWhat it costs at your volume

600 documents in both conditions is 1,200 calls at $0.000077 — about $0.09. The cost is linear; there is no batching and no caching.

Model choiceWhy this model

One model, one key — the kit standard. The fast tier was chosen because the task is short-form extraction with a fixed schema, and it reached 100.0% on clean input. Nothing here argues a bigger model would not do better on the scan; it was not run.

Other modelsThe same 120 calls, on other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

Both scored runs — 60 clean and 60 scanned. It excludes the 3-call probe, the 2-call token measurement, and the abandoned reasoning-on run, all of which are on the ledger but not in the published pair.

The codeEvery file, and what it is for

Four modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/corpus.pyCorpus generator — a swap seam

Writes 60 documents and their ground truth from a fixed seed, then emits each one twice — clean, and degraded by a documented set of OCR error classes.

You change it to: How badly the scan damages the page, and in which ways.

# The corpus: sixty one-page business documents, each emitted twice — clean, and degraded.

SEED = 20260814
N_DOCS = 60
CURRENCIES = ["USD", "EUR", "GBP"]
KINDS = ["INVOICE", "PURCHASE ORDER", "REMITTANCE ADVICE", "STATEMENT OF ACCOUNT"]
VENDORS = [
LABELS = {
ITEMS = [
def _money(rng):
def _date(rng):

evals/baseline.pyFree floor — a swap seam

Label-anchored rules with glyph un-confusion. Pure code, no key, and the honest half of the comparison.

You change it to: What the model is being compared against.

# The free floor: label-anchored rules over whatever text layer you were handed. No model, $0.00.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
ANCHORS = {
UNCONFUSE = str.maketrans({"O": "0", "o": "0", "l": "1", "I": "1", "S": "5", "B": "8", "|": "1"})
MONEY = re.compile(r"(\d[\d,]*\.\d{2})")
DATE = re.compile(r"(\d{4})[-/ ](\d{1,2})[-/ ](\d{1,2})")
CODE = re.compile(r"\b([A-Z]{2,4})[-\s]?(\d{4,6})\b")
CCY = re.compile(r"\b(USD|EUR|GBP)\b")
def _label_value(text, names):
def extract(text):

evals/run.pyExtractor

One model call per document, the same prompt in both conditions.

# The model run. One call per document, one condition per invocation.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
MAX_TOKENS = 700
def parse(text):
def main(argv):

evals/score.pyScorer — a swap seam

One scorer for both methods, splitting stated cells from not-stated cells so a misread is never averaged with an invention.

You change it to: What counts as correct.

# One scorer, used by the free floor and the model alike.

def load_gold(root):
def load_fields(root):
def norm(key, v):
def score_all(gold_rows, preds, fields):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 426 input and 63 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

A document is untrusted input. It is inserted into the prompt as data, and nothing the model returns is executed, evaluated, or used to build a path or a query. The reply is parsed as JSON and every value is a string that gets compared to a ground-truth string. There is no tool call, no shell, no database and no network egress beyond the single provider request.

The key is read from .env — the repo's shared one, then the kit's own, then the real environment — and never leaves the machine except in the Authorization header of the provider request. .env is gitignored from the first commit. ⚠︎ THE SHARED .env MEANS A BRAND-NEW KIT WITH NO .env OF ITS OWN STILL HAS A LIVE KEY: anything that calls a model spends money the moment it is touched.

The experimentWhat was actually run

Both conditions, both methods, one pass each. 120 model calls, 120 free-floor extractions, one scorer. Measured on 2026-08-14, runs r001-clean and r002-messy: 60 documents in two conditions, 120 live calls on one model. The structural gates below are code-checked; no injection rate is claimed, because none was measured.

Three of the four hold by construction rather than by a check that could fail. That is a weaker claim than a measurement and it is stated as such.

The resultThe kit reads documents and returns six fields. It never executes anything it reads, and the only thing it sends anywhere is the page you point it at.

Every cell scored in both conditions by both methods.

HonestyWhat this does not prove

• No prompt-injection resistance was measured — the corpus contains no attack.
• The refusal gate rests on 20 not-stated cells per condition. 100%% there means 20 of 20, which is not a strong claim.
• Neither run was repeated, so run-to-run variance is unmeasured.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Do not accept an extracted identifier from a scanned page without a check digit, a cross-reference, or a human eye. Everything else on the page survives the scan well enough to flow.

At the point the extracted record is written to whatever system consumes it.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• Not an OCR engine, and not a replacement for one.
• Not a check that the document is genuine, or that the numbers on it are right.
• Not a gate — it produces the evidence you would set a gate from.

WatchedWhat is watched, and why that one

The kit measures 23 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 16 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 60, extraction_cells 340, failures 0, refusal_cells 20) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 4 runs measured the same system — they differ on condition, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Per document.

In one lineNo framework, deliberately

There is no framework here and there is nothing for one to do. The kit is a loop over documents, one call each, and a scorer. No retrieval, no chaining, no agent, no state.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

It is a for-loop: read the file, build the prompt, one call, parse JSON, score. That is the whole of it, and a framework would add a dependency, a vocabulary and an abstraction over five lines that already read clearly.

The other sideWhat a framework costs you

• A dependency a forker must install to run a kit that currently needs nothing.
• An abstraction over a loop that is already readable end to end in one sitting.
• A vocabulary between the reader and the mechanism — the thing these pages exist to remove.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r002-messy on the fast tier, 2026-08-14. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b000-baseline, b001-baseline recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call, mess and all.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT. Granted by us because we wrote every byte of it — the generator is in the kit and its output is a pure function of seed 20260814. Nothing is derived from a third-party corpus, so there is no upstream licence to honour and no attribution owed. Verified 2026-08-14 by rebuilding from a clean tree and diffing byte for byte (tools/build_corpus.py --verify). Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-field exact match, both conditions

Is this cell's value the one the page states — or, where it states nothing, did the method correctly return nothing?

MethodHow the test was run

evals/score.py, in-process, no key. Gold comes from data/gold.jsonl, written by the corpus generator BEFORE the degrader runs — so both conditions are scored against one answer key and cannot drift apart.

The inputOne real row, seen by every grader

The formulaWhat it computes

norm(got) == norm(want), where norm collapses whitespace, strips trailing punctuation and lowercases; money reduces to two decimals; codes uppercase and de-space; currency truncates to three letters. Applied identically to gold and prediction, and to both methods. No stemming, no fuzzy distance, no substring credit.

IterationsFour rulers, four wrong numbers

The tell was the ranking inverting, not the value moving. Noise moves a number; an artifact reorders the comparison the page exists to make.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: this grader, against gold written by the corpus generator BEFORE the degrader ran — not against another model. Both conditions share one answer key, so the two halves of the comparison cannot drift apart.

How tight can the band be? There is no threshold to tune — the grader is == and has no knob. What has a denominator worth stating is the refusal set: 20 cells per condition, so ONE invention moves that rate by 5 points. Treat 100%% there as 20 of 20, not as a strong claim.

Cadence: Every paid run, and on any change to data/fields.json or to the degradation rates in src/corpus.py. Both change what is being asked or how damaged the page is, so neither can be compared across the change.

The decisionWhen to reach for it