The business caseThe problem this solves

A claims clerk or a front-desk coordinator opens a payer's eligibility portal, reads five things off the screen -- the member id, the plan, whether cover is active, the specialist copay and what is left on the deductible -- and types them into something else. Every payer's portal lays them out differently, none of them offers an export, and a copay typed one row out is quoted to a patient at the desk. reading five fields off a payer's eligibility screen and re-typing them by hand

The inputThe actual screenshots

The corpus is 60 screenshots, 6.10 MB (json 1 · jsonl 1 · md 1 · png 60). Every other OCR kit in this series reads a DEGRADED image -- a cheque photographed on a desk, a meter in bad light, a scan. A screenshot is machine-rendered: perfect pixels, no blur, no skew, no shadow, no sensor noise. That is the whole point of choosing it. It lets the kit ask a question none of the others can -- when the input is flawless, does the OCR stage buy you anything at all? -- and the answer turns out not to be about character accuracy at all. There is no added noise and no synthetic damage anywhere, because a screenshot with grain added is not a screenshot any more. The difficulty is real to screenshots instead: four layouts, two themes, three widths, two device scale factors, seven traps and eight cases where a scroll boundary cuts a value off.

The outcomeWhat a good result looks like

a five-field benefit record -- member id, plan name, coverage status, specialist copay and deductible remaining -- with a value the screen does not show left empty rather than filled in

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• You need a benefit record you can file without a person re-reading it — Mistral OCR 4.1 in front of the model
  99.67% of cells, indistinguishable from a perfect reading on this corpus, and it is the only one of the three arms that correctly leaves a scrolled-out value empty instead of lifting a word off the footer.
• Your portals are tables and definition lists, not cards — AWS Textract Detect Text
  its character accuracy is roughly 17x better and on table and deflist layouts it scores 0.9625 and 0.9857 against Mistral's 1.0000 and 0.9857 -- a difference of a cell or two, at less than half the price.
• You want to know whether to buy an OCR engine at all — run the free floor and the oracle arm first -- both cost nothing
  they bracket the question. The floor (0.7367) is what pure code gets; the oracle (0.9900) is what the model gets given a flawless reading. If your floor is already close to your oracle, no engine will earn its price.
• You are choosing an engine on its published character error rate — don't -- score the records, not the text
  this kit is a counter-example measured end to end: the engine with roughly 17x the character error rate wins by 17 cells to 1, p=0.0001.

At a glanceHow the whole thing runs

Run once, for real, on 2026-09-15. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-09-15 — r003-eligibility-screenshot-mistral. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Observed on this machine: a checkout with no .env and no key configured runs python3 -m evals.run and prints the whole five-arm table in about 1.4 seconds, because the screenshots, the answer key, the perfect-reading text, both engines' committed readings and every scored row are in the repository. python3 -m evals.score --self-test and python3 -m evals.run --self-test both pass with no key. The board serves the same committed run on port 9465 and is complete without a provider -- the 'empty' screenshot above is that state. What a clean checkout CANNOT do is buy a new reading: that needs a vendor credential, and re-rendering the corpus needs Chrome.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. Measured on the fast tier after Mistral OCR 4.1 — the model call only, timed around each of the 60 calls on the winning arm. It does NOT include the OCR request: stage 1 is a separate synchronous call to a vendor and this kit records its UNITS rather than its latency, so adding the two would produce a number describing neither stage. A product that must answer while somebody stands at a desk owes its own end-to-end figure and this is not it.

Current processWhat it replaces

reading five fields off a payer's eligibility screen and re-typing them by hand

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusCorpusCorpusCorpusCorpusEvalsEvalsArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• png
• jsonl
• json
• md
• 4 of 11 stations carry a recorded failure

scroll → 11 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one screenshot is

none -- all 60 screenshots are scored and there is no training, tuning or held-out set, because nothing here is fitted. The slices that matter are cut AFTER scoring, by layout, theme and trap, from metadata the corpus builder recorded.

SetupWhat the setup figure measured

There is no index and nothing is built. One screenshot is one page and goes whole into one call, so there is nothing to chunk, embed or retrieve -- the zero is the absence of a step, not a fast one.

LicenceLicence

MIT -- part of this repository, generated by it and owned by it. There is no upstream grant to honour because no third-party material is used; data/corpus/SOURCES.md records what is invented and why.

Bring your ownBring your own screenshots

Drop your own PNGs in data/corpus/ named <case_id>.png and write one line per case into data/gold.jsonl carrying fields (the five answers, with null where the screen does not show one), printed (the same values AS PRINTED, with currency symbols and separators -- this is what the propagation grader searches the reader's text for), reference_text (what a perfect reader would return) and a capture block naming layout, theme and trap so the slices still cut. Then run a station, then src/extract.py, then evals/run.py. Nothing else changes: arms are discovered from filenames.

⚠︎ And what stops being true when you do: reference_text is the field a real corpus cannot honestly supply, and it is the one every number in this kit's headline depends on. Here it is written by the generator from the same row model that produced the HTML, so the character error rates and the ORACLE arm are exact by construction. On your screenshots you would have to type it, which makes it an opinion -- and the oracle arm, the accuracy floor derived from it, and every stage-1 error rate would inherit that opinion. Field exact match against your own fields survives the move; the ceiling and the character error rates do not.

TracksWhich path you should run this on

This kit’s input is image, so it carries a capability stage in front of the model. The engine knows 5 arrangements, ruled 2 out here, and this kit runs A1 — measuring 2 paths through it against the same 60 screenshots and the same graders. Everything below is that run.

Cost is units × today’s price — the kit records minutes and pages, never dollars, so a vendor price move never invalidates what was measured and never costs a re-run.

FunnelHow the 2 were chosen

This is here so you can see the 2 were not cherry-picked — and so you can see where they disagree with the filters. 2 of the 2 this kit ran survive all 3 rules above.

Counts and rules only. The vendor names and list prices behind the 8 rows that survived these rules and were not run are estimates, and estimates live on the Admin catalogue, not on a kit page.

VolumeYour volume

The cost of one screenshot is measured: 60 pages over 60 screenshots (lenses/Data/corpus/doc_count), from this run’s own recorded units. Everything the buttons change is that number multiplied — no rate, no number and no ranking moves with the volume.

LensWhat matters most to you

A lens re-orders the table below and changes nothing in it. No cell value moves, no row is hidden, and a path below the floor of 0.9700 can never be promoted to first — hiding a row is how a preset manufactures a winner.

MeasuredThe 2 paths this kit ran

cheapest path clearing the floor of 0.9700: $0.240000 for the whole scored run

ArrangementsThe 5 arrangements

Eligibility, the numbers each arrangement owes, the artifact it produces and whether it can attribute an error are structural — they are true of an arrangement whether or not it ran, and they are read from the standard rather than typed here. The cost and accuracy rows are blank for every arrangement this kit did not run, because nothing measured them.

Where to spendThe number that tells you where to spend

A2 and A3 cannot produce that sentence at any price: one supplier, one number, no span to point at. That is the entire argument for the arrangement this kit runs, and it is the reason A1 is the only one of the 5 that owes a propagation figure at all.

StabilityWhere the ranking would flip

Every condition above is arithmetic over the prices and the scores on this page. None of it is a forecast: it says what would have to be true, not what will be.

ScopeWhat is on this page, and what is not

Re-runRun it on your own stack

Point it at your own screenshots. Drop your own PNGs in data/corpus/ named <case_id>.png and write one line per case into data/gold.jsonl carrying fields (the five answers, with null where the screen does not show one), printed (the same values AS PRINTED, with currency symbols and separators -- this is what the propagation grader searches the reader's text for), reference_text (what a perfect reader would return) and a capture block naming layout, theme and trap so the slices still cut. Then run a station, then src/extract.py, then evals/run.py. Nothing else changes: arms are discovered from filenames.

Re-scoring costs nothing. All 3 graders are deterministic code with a known key — Field exact match, Stage-1 character and word error, Propagation — so a recorded run can be re-scored as often as you like and returns the same answer. The only thing you pay for is the transcription and the model call themselves.

Before you spend anything. Observed on this machine: a checkout with no .env and no key configured runs python3 -m evals.run and prints the whole five-arm table in about 1.4 seconds, because the screenshots, the answer key, the perfect-reading text, both engines' committed readings and every scored row are in the repository. python3 -m evals.score --self-test and python3 -m evals.run --self-test both pass with no key. The board serves the same committed run on port 9465 and is complete without a provider -- the 'empty' screenshot above is that state. What a clean checkout CANNOT do is buy a new reading: that needs a vendor credential, and re-rendering the corpus needs Chrome.

Swap the stage-1 station. The arrangement and the graders are unchanged by a swap, so your re-run stays comparable to this one. 10 rows in the Document / image to structure stage carry a price that both parses and can be re-read today: AWS Textract Detect Text (the one this kit runs), AWS Textract Queries, AWS Textract Tables, AWS Textract Forms, Google Document AI Enterprise OCR, Google Document AI Layout, Google Document AI Form Parser / Custom Extractor, Mistral OCR 4.1 (the one this kit runs), Mistral OCR 4.1 batch, Reducto.

Before you use theseRead this before you use these numbers for your own case

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 2,572 characters assembled, 1,747 are instructions — 68% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Replayed from the kit's own src/prompt.py::build for ES-0012, the sample clip, against that case's committed stage-1 text -- so the decomposition cannot be a tidy description that drifted from what was sent. It is checkable: build() itself asserts every part occurs in the assembled user prompt in ascending position and raises if one does not, the run records carry the assembled prompt's sha256 prefix (99cf4377cb423004) and the per-part character counts, and those counts match the parts published here (task 72, shape 203, reading 550). Per-part TOKEN counts are absent rather than estimated: the provider reports one figure for the whole request and splitting it would be arithmetic wearing a measurement's name.

What the model receivesWhat the model actually receives after stage 1

Read from results/ocr-*.jsonl, the committed output of both purchasable stations and the perfect reader over all 60 screenshots. Character and word error are measured against the text the corpus builder laid into the pixels.

Full promptThe prompt in full

You read the text of a screenshot taken from a health-insurance payer's eligibility portal and return one benefit record as JSON.

Return EXACTLY these five keys and no others:
  member_id                 the member's own identifier, copied character for character
  plan_name                 the benefit plan's printed name
  coverage_status           the eligibility status exactly as the portal words it
  specialist_copay_usd      the SPECIALIST visit copay, as a plain decimal number
  deductible_remaining_usd  the deductible still OUTSTANDING, as a plain decimal number

Rules:
1. USE null WHEN THE SCREENSHOT DOES NOT SHOW THE VALUE. A portal pane can be scrolled, so a line may be missing from the picture entirely. null is the correct answer for a value that is not there. Do not infer it, do not carry it over from another line, do not guess.
2. IF A VALUE IS CUT OFF, RETURN WHAT IS ACTUALLY PRINTED. A value truncated by a column edge is a shorter string, not a wrong one. Do not complete it.
3. COPY CHARACTERS EXACTLY. Portal type makes 0 and O, 1 and I and l, 5 and S, 8 and B, 2 and Z, 6 and G hard to tell apart. Return what the text gives you. Do not normalise a member id into what you think it ought to be.
4. WATCH THE NEAR-MISSES. 'Deductible' is not 'Deductible Remaining'. 'Primary Care Copay' is not 'Specialist Copay'. Take the value that belongs to the label you were asked for.
5. LABELS AND VALUES MAY BE FAR APART, OR IN TWO COLUMNS THAT INTERLEAVE. Pair a value with the label it belongs to, not with the label nearest it in the text.
6. MONEY IS A PLAIN DECIMAL: no currency symbol, no thousands separators. $1,250.00 becomes 1250.00.
7. Return the JSON object alone. No explanation, no code fence, no commentary.

Read this eligibility screenshot's text and return the benefit record.

Return exactly this shape, with your values in place of null:
{
  "member_id": null,
  "plan_name": null,
  "coverage_status": null,
  "specialist_copay_usd": null,
  "deductible_remaining_usd": null
}

----- SCREENSHOT TEXT -----
# Harbourstone Health

Eligibility > Member Search > Benefit Detail

## Eligibility & Benefits

Member ID

VXH799737493

Group #

GRP-252060

Plan

Alliance HMO 2500 Tier I

Plan Year

2026

Rendering NPI

8539989503

Coverage Status

Pending

Effective

12-Jun-2026

Deductible

$250.00

Deductible Remaining

$0.00

Out-of-Pocket Max

$798.00

Specialist Copay

$15.00

Primary Care Copay

$50.00

Emergency Room Copay

$500.00

Verified 09/14/2026 10:21 ET · This summary is not a guarantee of payment.
----- END -----

Raw responseThe raw response

{
  "member_id": "VXH799737493",
  "plan_name": "Alliance HMO 2500 Tier I",
  "coverage_status": "Pending",
  "specialist_copay_usd": 15.00,
  "deductible_remaining_usd": 0.00
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Turn a screenshot of a payer eligibility portal into a structured benefit record — 60 screenshots. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Every grader is pure code: string equality, a decimal comparison and a Levenshtein distance. No model grades anything, so re-scoring a committed run costs $0.00 and returns the same answer every time. A null gold answered null is a HIT and a null gold filled in is its own verdict, invented -- folding those together would hide the failure that gets a claim denied.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar here is a MEASURED token count multiplied by a PUBLISHED rate -- a projection onto a rate card, not a bill anybody paid. The token counts are the provider's own, from the usage block on each of the 180 calls across the three paid arms. ⚑ THE CARD IS NOT THE PROVIDER THAT RAN THE CALLS: this site withholds the runtime provider, so the run's own tokens are priced against published lists anybody can read. ⚑ AND THE OCR STATIONS CARRY NO DOLLAR AT ALL, ON PURPOSE: a kit records UNITS. 60 pages went to each station and the price is applied at build time from build/facts/capabilities.json, so a vendor repricing never invalidates this run and never costs a re-run.

Rates checked 2026-09-12.

Grading unitWhat the grading figure prices

Free is the grader's OWN unit, not the pipeline's. The extraction costs tokens and the OCR stage costs pages; the ruler costs nothing, which is why the whole eval can be re-run against the committed files by anyone who clones this repo with no key configured.

The gradersThree ways to grade

The floor is run on the ORACLE's text, not on an engine's, so it is being given a perfect reading and still loses. That is deliberate: a floor handicapped by a bad reader would flatter the model.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — top_k, chunk size, prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-09-12. Every dollar here is the model call only: this kit also runs Mistral OCR 4.1 before the model, and one screenshot costs $0.00423 end to end — see the Tracks panel. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

Grading is arithmetic and costs nothing. python3 -m evals.run re-scores all five arms off the committed files in about 1.4 seconds with no provider reachable from that module at all, so there is no way to re-buy a call from it. The second bill a forker pays is at stage 1, not here: re-running the eval is free, re-running the CORPUS means buying 60 pages again on each engine.

Cost driversWhat actually moves the bill

• Pages at the OCR stage, and it is the whole decision. Each arm bought 60 pages. At today's list prices that is $0.24 on Mistral OCR 4.1 and $0.09 on AWS Textract Detect Text, against $0.014-$0.114 for all 60 model calls across the 5 published cards -- so even at the low end of that band stage 1 is roughly six to seventeen times the model bill, and choosing the engine IS choosing the cost.
• How much text the portal page carries, not how the prompt is worded. Input averages 740 tokens per call of which 2022 characters are the fixed instruction and contract that never vary; the rest is the reading.
• Output is negligible and bounded: 70 tokens on average, 76 at the worst reply, against a ceiling of 1000 that nothing came near.

Your volumeWhat it costs at your volume

Linear in both stages at this size. Nothing batches or caches, no tier boundary is crossed, and neither station has a volume ladder that bites at hundreds of pages. The first thing that changes shape is not the price but the call pattern -- see Architecture.breaks_at_scale -- and the first real saving available is a batch product neither arm used.

Model choiceWhy this model

One fast tier, one key, reasoning disabled -- because the model is deliberately NOT the variable. This kit changes the OCR station between arms and holds everything downstream byte-identical: the same prompt (sha 99cf4377cb423004 on every run record), the same settings, the same ceiling. A second model tier would have made the comparison two-dimensional and answered neither question.

Other modelsThe same tokens, five price lists

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

the whole 60-screenshot scored run on the winning track: 44412 input tokens and 4203 output tokens, with reasoning explicitly DISABLED on every call.

The codeEvery file, and what it is for

Nine modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pythe screenshots — a swap seam

60 PNGs rendered by headless Chrome from generated values, in four layouts, two themes, three widths and two device scale factors -- and the answer key is written from the values that produced the HTML, never read back off the pixels

You change it to: the seed, the layouts, the traps and the clipping geometry. Changing the seed changes dataset_version and no published number carries across

# UC0465 -- generate the eligibility-portal SCREENSHOT corpus, gold exact BY CONSTRUCTION.

HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.dirname(HERE)
DATA = os.path.join(ROOT, "data")
CORPUS = os.path.join(DATA, "corpus")
HTML = os.path.join(DATA, "_html")
SEED = 465
N_CASES = 60
DATASET_VERSION = "eligibility-screenshot-v1-%d-cases-seed%d" % (N_CASES, SEED)
FIELDS = ["member_id", "plan_name", "coverage_status",

src/stations/textract.pythe OCR station

a plug: SLUG, STATION, PAID, UNIT and read(). src/stations/ DISCOVERS modules rather than listing them, so an engine is added by dropping in one file

# STATION: AWS Textract DetectDocumentText. PAID, one call per screenshot.

SLUG = "textract"
STATION = "AWS Textract Detect Text"
PAID = True
UNIT = "pages"
HOW_TO_RUN = ("python3 tools/run_textract.py --limit 1
ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

src/stations/mistral.pythe second OCR station

the same plug against a different vendor -- the only thing that differs between arms

# STATION: Mistral OCR 4.1. PAID, one call per screenshot.

SLUG = "mistral"
STATION = "Mistral OCR 4.1"
PAID = True
UNIT = "pages"
HOW_TO_RUN = ("python3 tools/run_mistral_ocr.py --limit 1
ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

tools/run_textract.pythe paid runners

the only files that spend money at stage 1, deliberately separate from the station modules so that an import can never buy a page. Hand-written SigV4, no boto3

# UC0465 stage-1 STATION: AWS Textract DetectDocumentText, one call per screenshot.

HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.dirname(HERE)
DATA = os.path.join(ROOT, "data")
CORPUS = os.path.join(DATA, "corpus")
RESULTS = os.path.join(ROOT, "results")
ENV = os.path.join(os.path.dirname(os.path.dirname(ROOT)), ".env")
STATION = "AWS Textract Detect Text"
STATION_SLUG = "textract"
RUN_ID = "s001-eligibility-screenshot-textract"

src/prompt.pythe prompt

one module constant and a build() that takes ONE argument -- the station's text. Stage 2 never sees the image, the layout, the trap or the gold

# SEAM 2 -- what the model actually receives, assembled once and published in full.

FIELDS = ["member_id", "plan_name", "coverage_status",
SYSTEM = (
SHAPE = json.dumps({f: None for f in FIELDS}, indent=2)
def build(text):
def render(parts):
def fixed_prefix_chars():
def main(argv=None):

src/extract.pythe model call

one urllib POST per screenshot, reasoning explicitly disabled, max_tokens 1000

# SEAM 3 -- stage 2. ONE model call per screenshot's stage-1 text.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
RESULTS = os.path.join(HERE, "results")
SLUG = "eligibility-screenshot"
THINKING = THINKING_OFF
MAX_TOKENS = 1000
def _gold():
def _widest_reply_chars():
WIDEST_REPLY_CHARS = _widest_reply_chars()

src/floor.pythe free floor

label-anchored regular expressions and nothing else -- the cheapest thing that could possibly work, so the model has something to beat

# THE PURE-CODE FLOOR -- what four lines of regex get you, so the model has something to beat.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
RESULTS = os.path.join(HERE, "results")
PATTERNS = {
MONEY_FIELDS = ("specialist_copay_usd", "deductible_remaining_usd")
def extract(text):
def main(argv=None):

evals/score.pythe graders

field exact match, character/word error against the reference text, and propagation. All pure code: a re-score costs nothing and returns the same answer every time

# UC0465 graders -- field exact match, stage-1 error, and propagation. ALL PURE CODE.

HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.dirname(HERE)
DATA = os.path.join(ROOT, "data")
RESULTS = os.path.join(ROOT, "results")
KIT = "eligibility-screenshot"
ARCHITECTURE = "A1"
STAGE = "doc_image_to_structure"
ORACLE_STATION_FILE = "ocr-oracle.jsonl"
MONEY_FIELDS = ("specialist_copay_usd", "deductible_remaining_usd")

src/app.pythe board

a stdlib HTTP server over the committed run on port 9465. do_POST is not defined, so a write is answered 501 without a line of code choosing to

# UC0465 -- the eligibility-screenshot review board. A stdlib HTTP server, port 9465.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
CORPUS = os.path.join(HERE, "data", "corpus")
PORT = 9465
TITLE = "UC0465 eligibility-screenshot"
def resolve_port(argv=None):
CFG = config.load()
def _arms():
ARMS = _arms()

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 740 input and 70 output tokens per query at top-k 1, of which about 150 tokens are fixed (system prompt, chat framing, the question itself) and the rest is retrieved context. Change the retrieval depth and the input scales with it. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model and top-k follow the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

⚠︎ NOT FIRED. evals/injection.py exists in this kit and no run of it is on disk -- no result file, no trial count, no rate. This section publishes that absence rather than a number, because the alternative is a threat panel that reads as reassurance. What IS structural and can be stated without a run: the answer contract names five keys and none of them is an action; src/extract.py takes the first JSON object and discards prose around it; and the board has no write route at all. The exposure is specific and worth stating plainly: stage 2 never sees pixels, only text lifted off a picture, and that text carries whatever was PRINTED on the page -- including a portal's free-text notes and remarks fields. Anyone who can get a string onto that page can get a string into this prompt, and the OCR engine is not in the business of refusing to read something. That is a real door and it has not been knocked on.

No credential is in this repository and none ever has been. The kit reads one gitignored .env, with a per-kit override: the model key reaches exactly one module (src/adapters/) and each OCR credential reaches one tool under tools/. No key is sent to the browser -- the board is handed a display string and never MODEL, enforced by src/config.py::MODEL_DISPLAY rather than by template discipline. AWS names the caller's account id and IAM user in an AccessDeniedException; that identity is REDACTED before it is printed or written, so a run file can be committed and the kit handed on.

The experimentWe did not attack it — and here is exactly what that leaves open

The probe is authored and unrun. If it is run, the payload route is the one that matters on this kit and it is not the usual one: there is no markup, no invisible character and no file format here, because what reaches the model is OCR output. A payload has to be legible text printed on a screen and then read back by an engine -- which means it must survive the reader, and the reader is the variable this kit is measuring. An attack that works after one engine may not survive the other, so the honest probe runs on both arms. Both were run for real on no run — results/ holds no eval-x001-eligibility-screenshot-injection file, and no adversarial figure appears anywhere in this spec. evals/injection.py is committed and has never been executed.

The resultThe adversarial probe is WRITTEN AND HAS NOT BEEN RUN. There is no result file and no rate to publish.

no trial has been run. evals/injection.py is committed and there is no results/eval-x001-* file for it.

HonestyWhat this does not prove

• every rate a threat panel would normally carry -- none was measured, because the probe was not run
• whether a payload printed into a portal's free-text field survives either OCR engine intact
• whether the two engines differ in what they pass through, which on this kit is the interesting question and the one only a two-arm probe could answer
• anything about a portal that renders attacker-controlled text as an image

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

NOTHING THIS KIT PRODUCES IS A BENEFIT DETERMINATION. The output is a READING of a screen: five values a person can check against the same picture in a second. It never quotes a benefit to a member, never attaches anything to a claim, never writes to a claims or practice management system, never authorises a service and never tells anyone they are covered. A benefit record is a proposal a human verifies, and this kit does the first half only.

src/prompt.py::FIELDS is the whole answer contract and it names exactly five keys -- none of them an action -- so an instruction to act has nowhere to be written even when one is printed on the page. src/extract.py takes the first JSON object and nothing else; prose around it is discarded rather than obeyed. src/app.py serves read-only JSON off disk on port 9465 and has NO write route at all -- do_POST is not defined, so a POST is answered 501 without a line of code choosing to. src/config.py ships with no save()/WRITABLE, deliberately unlike the kit it was carried from, so there is not one route on the server that puts bytes on disk.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT an eligibility determination, a coverage guarantee or a quote. Every screenshot in this corpus carries the payer's own disclaimer -- 'This summary is not a guarantee of payment' -- and neither does this.
• It is NOT a defence against a portal that displays something wrong. Every measurement asks whether the record matches what the SCREEN SHOWED; nothing anywhere asks whether the screen was right.
• It is NOT tested against adversarial input. evals/injection.py exists and has not been run. The concern is concrete: stage 2 never sees pixels, only text lifted off a picture, and that text carries whatever was printed on the page -- including a portal's free-text notes. Anyone who can get a string onto that page can get a string into this prompt.
• It is NOT measured on real screenshots, so nothing here says what happens to a capture with a cursor on a digit or a dialog over a copay.

WatchedWhat is watched, and why that one

The kit measures 23 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

12 measured by the latest run 11 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (capability.architecture A1, capability.stage doc_image_to_structure, documents 60, failures 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

5 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 9 chips that all say so.

extraction · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 9 chips that all say so.

extraction · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on capability.station, capability.units — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 5 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Every grader in this kit is arithmetic over data/gold.jsonl and touches no network, so the whole board except the paid columns re-derives on every commit for $0.00 -- python3 -m evals.run, python3 -m evals.score --self-test and python3 -m evals.run --self-test all run with no key and no cloud credential configured, in about a second and a half. The part that is NOT free to re-run is stage 1: re-reading the corpus is 60 pages on each engine, and that is the only reason to re-run it -- a vendor changing the model behind a hosted endpoint would move the character error rates without moving anything in this repository. The red team is not free either and has never been run at all.

In one lineNo framework, deliberately

No agent framework, no orchestration library, no vector store, no schema library and no cloud SDK -- and the reason is measurable rather than ideological. There is ONE model call per screenshot with no tools, no retrieval, no memory and no branching, and the only thing in front of it is an OCR request. Every import is the Python standard library, AWS SigV4 included. There is no pip install at all for the pipeline or the eval layer; Node is needed only to take the UI screenshots and Chrome only to regenerate the corpus.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

screenshot -> [OCR station, one per track: src/stations/*.py] -> text reference text -> [no engine at all: src/stations/oracle.py] -> text text -> [one model call: src/extract.py] -> {member_id, plan_name, coverage_status, specialist_copay_usd, deductible_remaining_usd} text -> [pure code: src/floor.py] -> the same five fields by regex, $0.00 every arm -> [pure code: evals/score.py] -> field exact match, stage-1 error, propagation

The other sideWhat a framework costs you

• SigV4 is about eighty lines in this repository that boto3 would have provided. That is the price of the no-install claim and it is paid once.
• No structured-output mode means a malformed reply is possible. That is deliberate -- it is measured rather than prevented -- but a production system would probably want the guarantee.
• No retry framework beyond the four retries in src/adapters/. A vendor outage stops the run, and --resume is what makes that survivable rather than anything clever.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r003-eligibility-screenshot-mistral, 2026-09-15. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

1 run not plotted. b000-eligibility-screenshot-floor-oracle recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, AWS_ACCESS_KEY_ID, AWS_REGION, AWS_SECRET_ACCESS_KEY, BASE_URL, CHROME, FOUNDRY_RUN_ID, MAX_CALLS_PER_DAY, MISTRAL_API_KEY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- part of this repository, generated by it and owned by it. There is no upstream grant to honour because no third-party material is used; data/corpus/SOURCES.md records what is invented and why. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineField exact match

did this cell come back with the value the screenshot actually shows

MethodHow the test was run

evals/score.py::cell_verdict over every gold cell -- 60 screenshots x 5 fields = 300 -- never over what an arm happened to return. A reply that omits a field scores that field wrong, because a benefit record with a hole in it is wrong. Normalisation is per field and argued: money as a decimal to two places, coverage_status case-folded (a closed vocabulary), member_id and plan_name CASE-SENSITIVE and character-exact, because 1/I/l is the trap and a case-folding comparison would silently delete it.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

One screenshot, two readers, one cell of difference -- and the cell is a copay that gets quoted to a patient at a desk. Textract's text lists all three copay LABELS before any of the three AMOUNTS, so the model pairs 'Specialist Copay' with the wrong figure and returns 50 where the gold is 15.00. Mistral's text interleaves label and value, and the record comes out right. Note what the graders disagree about: on character accuracy Textract is nearly perfect on this page and Mistral is not, and the propagation grader charges the failure to stage 2 because the string '$15.00' IS in Textract's text -- just three lines away from its label.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, written from the values that generated the HTML, so the answer key is not an opinion.

No true/false rates for this grader. It records 4 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? 0.97 is the capability floor, derived from the oracle arm's 0.9900 and not from either track.

Cadence: on every run; re-scoring a committed run is free

The decisionWhen to reach for it

In one lineStage-1 character and word error

how far the reader's text is from what was actually printed on the page

MethodHow the test was run

evals/score.py::cer and ::wer -- Levenshtein over a case-folded, punctuation-stripped, whitespace-collapsed basis, against reference_text. ABSENT for the oracle arm, which ran no engine: 'perfect' and 'not measured' must never print as the same number.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

One screenshot, two readers, one cell of difference -- and the cell is a copay that gets quoted to a patient at a desk. Textract's text lists all three copay LABELS before any of the three AMOUNTS, so the model pairs 'Specialist Copay' with the wrong figure and returns 50 where the gold is 15.00. Mistral's text interleaves label and value, and the record comes out right. Note what the graders disagree about: on character accuracy Textract is nearly perfect on this page and Mistral is not, and the propagation grader charges the failure to stage 2 because the string '$15.00' IS in Textract's text -- just three lines away from its label.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl reference_text -- the text the corpus builder laid into the pixels.

No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? there is no pass mark; the number is only meaningful beside the field exact match it did or did not predict

Cadence: whenever a station is re-run

The decisionWhen to reach for it

In one linePropagation

of the cells that came out wrong, how many had already lost their evidence in the reader's text

MethodHow the test was run

evals/score.py::evidence_lost and ::propagation -- the PRINTED form of the value ($1,250.00, not 1250.00) is searched for in the station's text. Blunt on purpose: it cannot be argued with and it never credits a station for text it did not return. The denominator is WRONG cells, not all cells, and the rate is null when nothing was wrong -- undefined, never zero. A null cell can never charge stage 1: there is nothing to lose.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

One screenshot, two readers, one cell of difference -- and the cell is a copay that gets quoted to a patient at a desk. Textract's text lists all three copay LABELS before any of the three AMOUNTS, so the model pairs 'Specialist Copay' with the wrong figure and returns 50 where the gold is 15.00. Mistral's text interleaves label and value, and the record comes out right. Note what the graders disagree about: on character accuracy Textract is nearly perfect on this page and Mistral is not, and the propagation grader charges the failure to stage 2 because the string '$15.00' IS in Textract's text -- just three lines away from its label.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl printed -- the values as they appear on the screen, symbols and separators included.

No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? 0.6 and 0.2 are the thresholds evals/score.py::propagation uses to word its own reading; they are editorial bands, not pass marks

Cadence: on every scored run

The decisionWhen to reach for it