The business caseThe problem this solves

A correction to a previously published rule states, in its own abstract, that something changed — but not whether a regulated party needs to act differently or whether the wording was only cleaned up. Read closely, a compliance team can tell one from the other; unread, a possibly-material correction sits in the same queue as a typo fix until someone opens it. A compliance reviewer opening a Federal Register correction and reading it against the rule it corrects, side by side, to decide whether anything a regulated party must do has actually changed.

The inputThe actual documents

The corpus is 79 documents, 0.06 MB (txt 79). The pairing is the publisher's own: a Regulation ID Number tracks one regulatory action through its life, and the API can be searched by it directly (verified live, 2026-08-06 — see tools/fetch_corpus.py's module docstring for why the fields that sound like they'd do this, correction_of/corrections, do not). No citation-parsing, no guessing which two documents belong together.

The outcomeWhat a good result looks like

Each pair receives a materiality verdict — or an honest abstention or escalation — with a one-sentence reason, at a median of about 3.3 seconds per pair on the fast tier.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• No publisher-assigned label exists for the judgment you need — agreement between two independently run models
  there is nothing to score == against, and two independent readers agreeing is real signal even with no ground truth behind it — the same logic inter-rater reliability runs on outside AI.
• A gold label DOES exist for what you're judging — a gold-match grader, like docs-route's == against the Federal Register's own type field
  a real answer is available and should be used.
• You need to triage a queue of correction pairs before deciding whether a model is worth paying for at all — the free regex baseline (evals/baseline.py) first
  no key, no cost, and on this corpus it already tells you something: it agrees with the fast tier only 40.0% of the time and the reasoning tier only 53.8% of the time, well below the 64.7% the two paid models agree with each other.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-07. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-07 — r001-docs-redline-flash. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Clone, python -m tools.fetch_corpus, python -m tools.build_corpus, python -m evals.run --run-id b000 --baseline regex. Three commands, no key, no cost, and a scored baseline at the end of it.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. the model call only, one per pair, 40 pairs run serially. Excludes tools/fetch_corpus.py and tools/build_corpus.py (free, run once, not part of a graded call) and src/diff.py's alignment (pure code, sub-millisecond).

Current processWhat it replaces

A compliance reviewer opening a Federal Register correction and reading it against the rule it corrects, side by side, to decide whether anything a regulated party must do has actually changed.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusEvalsEvalsArchitectureArchitecturePromptPromptPromptPromptArchitectureArchitectureApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 9 stations carry a recorded failure

scroll → 9 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one document is

none — same reasoning as docs-route: a Federal Register title+action+abstract is short enough to send whole, so there is no chunker and no packer.

SetupWhat the setup figure measured

There is no index. src/diff.py's alignment step does the only preparation work, and it runs at call time over the two versions already on disk, not as a separate build stage.

LicenceLicence

Public domain. Federal Register documents are edicts of government and works of the United States Government, not subject to copyright protection in the United States (17 U.S.C. §105). The Office of the Federal Register places no restriction on reuse of the documents or of the API's output. Verified against federalregister.gov/developers/documentation/api/v1 on 2026-08-06, same verification docs-route already carries for the same API.

Bring your ownBring your own documents

Point tools/build_corpus.py at your own paired documents and write your own data/pairs.jsonl — one line per pair, {pair_id, v1_id, v2_id}. Nothing else changes: src/diff.py, src/prompt.py and the scorer all work on plain text, not on anything specific to Federal Register formatting.

⚠︎ And what stops being true when you do: This kit's measured result is about documents at title+action+abstract length, not full legal text. A pair whose substance differs only in body text neither of these fields carries will show no detectable change and is skipped, not measured as 'no change'.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 1,541 characters assembled, 1,118 are instructions — 73% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Rebuilt verbatim by src/prompt.build() over pair 2024-07437__2026-12700's aligned span — the same function the run called, not a transcription.

Full promptThe prompt in full

You are reviewing a correction to a regulatory document. You are given the original wording and the corrected wording of the one place where they differ. You do not summarise the document, you do not advise on it, and you do not explain the regulation — your only job is whether this specific change is material or editorial.

Classify this one change.

LEVELS:
  "material"  (Material) — Changes an obligation, a deadline, a number, a legal citation, or an amount — something a regulated party would act on differently. Missing one of these is the expensive direction: a compliance deadline that quietly moved and nobody re-read the notice.
  "editorial"  (Editorial) — Wording, formatting, punctuation, or an internal cross-reference — corrects how the rule reads without changing what it requires. Flagging one of these as material wastes a reviewer's morning on a typo.
  "unsure" — you are not confident enough to choose. A person will read it.

ORIGINAL WORDING:
Americans and to reflect recommendations from the National Academies of Science, Engineering, and Medicine while promoting nutrition security and equity and considering program administration. The changes are intended to provide WIC participants with a wider variety of foods that align with the latest nutritional science; provide

CORRECTED WORDING:
(nothing — this text was removed by the correction)

Answer with JSON only, in this exact shape:
{"materiality": "<one key from the list above>", "confidence": <0.0 to 1.0>, "why": "<one short sentence, at most 20 words>"}

Raw responseThe raw response

{"materiality": "editorial", "confidence": 0.95, "why": "Removal of introductory purpose language does not alter any obligation, deadline, number, or legal requirement."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check which federal rule corrections change your compliance duties — 40 pairs drawn from 79 real documents. Two tiers of one model family answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

Agreement between two independently run models on the same aligned span, scored by exact string comparison of their materiality verdicts — not a gold match. No publisher grades a correction's materiality, so there is nothing to score == against; see evals/score.py's module docstring for the full reasoning. Disagreement between the two runs is the kit's actionable output, not an error. A free regex baseline (evals/baseline.py) is scored into the same agreement comparison as the two models, exactly as docs-route's keyword baseline sits beside its model runs — the question is not just 'do two models agree' but 'does a model agree with something free any more than it agrees with itself'.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

build/facts/models.json in the Foundry repo — the same cards docs-route prices against. Token counts are the provider's own usage block, never an estimate; only the price per token comes from the card.

Rates checked 2026-08-07. The provider that actually ran r001 and r002 is not priced here. Its rate is not committed anywhere in this repo, and a dollar figure whose card cannot be named is exactly what this standard forbids.

Grading unitWhat the grading figure prices

The grader (evals/score.py::agree()) is a pure comparison of two already-computed verdicts — offline, no key, no cost. The pipeline it scores is NOT free; that cost is priced in Cost, per pair, the same distinction docs-route draws between its grader and its pipeline.

The gradersOne way to grade, and why it is the only one

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-07. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingScoring is free here, and it is a measured zero

evals/score.py::agree() is a pure comparison of two already-computed verdicts — offline, no key, no call. The regex baseline it is scored against is free the same way.

Cost driversWhat actually moves the bill

• Calls that hit the 1,200-token output ceiling and return nothing — 4 of 40 fast-tier calls and 1 of 40 reasoning-tier calls spent the whole ceiling reasoning, at zero return.
• The reasoning pass itself, enabled by provider default on every call — the larger share of output tokens on both models; see LLM.token_share_caption.
• The aligned span's length — the only per-pair variable in the prompt; the levels block (624 of ~1,540 prompt characters) is fixed overhead paid on every call regardless of span size.

Your volumeWhat it costs at your volume

Linear — one call per pair, no retrieval, no shared index to amortise. 400 pairs on the fast tier projects to about $0.078 on the Gemini-priced illustrative card (10 x the measured $0.0078 for 40), and roughly ten times the measured wall clock, serially, because nothing here runs concurrently.

Model choiceWhy this model

The fast tier was the default the shared .env already pointed at; the reasoning tier was run because the kit standard requires two independently run models for the agreement score — one model is a result, not a comparison.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The work behind every number on these pages: 40 aligned pairs from 79 Federal Register documents, one materiality verdict each. Flash's run scored all 40 — the fast tier's numbers are what this table prices, the same row Cost.cost_by_model[0] already uses.

The codeEvery file, and what it is for

Eight modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/fetch_corpus.pyFetch

two API queries per candidate — find corrections, then look up each one's RIN — into data/_fetched/

# Pull document PAIRS from the Federal Register API into data/_fetched/. Costs nothing, no key.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FETCHED = os.path.join(HERE, "data", "_fetched")
API = "https://www.federalregister.gov/api/v1/documents.json"
FIELDS = ["document_number", "title", "abstract", "action", "type", "publication_date",
PAUSE_SECONDS = 1.0
UA = "adhana-foundry-kits/UC0005-docs-redline (https://github.com/adhana-ai/adhana-foundry-kits)"
def _get(url):
def _search(params):
def find_corrections(want, page_size=100):

tools/build_corpus.pyCorpus boundary

writes the two document versions a reader sees as separate .txt files, and the pair manifest

# Turn a raw fetch into the shipped corpus: one .txt per document version, a manifest of pairs.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FETCHED = os.path.join(HERE, "data", "_fetched")
CORPUS = os.path.join(HERE, "data", "corpus")
PAIRS_FILE = os.path.join(HERE, "data", "pairs.jsonl")
MANIFEST = os.path.join(CORPUS, "manifest.json")
MIN_ABSTRACT = 80
def _clean(s):
def _document_text(rec):
def build():

src/diff.pyAlign

the pure-code word-level diff that finds the one changed span before the model sees anything

# Align two document texts and return the candidate changed spans. Pure code, no model.

def _tokens(text):
def align(v1_text, v2_text, min_chars=6):
def primary(v1_text, v2_text):

src/materiality.pyThe levels — a swap seam

material vs editorial, and what calling it either one costs to get wrong; every other module reads them from here

You change it to: the materiality categories themselves — change LEVELS and ORDER and the prompt and scorer follow, because neither holds a second copy

# What a flagged change means, and what routing it triggers.

LEVELS = {
ORDER = ["material", "editorial"]
ABSTAIN = "unsure"
def label_of(key):
def meaning(key):
def labels():

src/prompt.pyPrompt and parse

builds the prompt from the one aligned span, and parses the reply into four distinct states

# Build the one message pair that asks for a materiality verdict on an already-found change.

SYSTEM = (
def _level_block():
def build(span):
def parse(text):

src/redline.pyThe call, and the floor

the one model call, and the confidence floor that turns a low-confidence verdict into an escalation

# The AI layer: two document versions in, one materiality verdict out. The whole model surface.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 1200
CONFIDENCE_FLOOR = 0.6
def pairs():
def load_version(doc_id):
def decide(cfg, v1_text, v2_text, complete=None, floor=CONFIDENCE_FLOOR):
def outcome(rec):
def summary_line(pair_id, rec):

evals/baseline.pyThe free baseline — a swap seam

a regex over the changed span's surface shape — no model, no key

You change it to: the free baseline. The two regex lists are thirty-odd lines; a fairer or harsher baseline is an edit to them

# The zero-cost baseline a materiality model has to beat. No model, no key.

def regex_predict(span):
BASELINES = {"regex": (regex_predict, "%d regex(es) over the changed span's surface shape"

evals/score.pyScorer

agreement between two runs, not a gold match — see the module docstring for why

# Score materiality verdicts by AGREEMENT, not by a gold match. Pure arithmetic — no judge.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
def pair_ids():
def outcomes(records):
def agree(records_a, records_b, name_a, name_b):
def agreement_text(a):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 428 input and 380 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

A local, single-user tool over public Federal Register document pairs. It binds 127.0.0.1, has no auth, no rate limit and no session — correct for a demo on your own machine and wrong for anything else. There is no chat box anywhere in it: the only text a stranger controls is the corrected wording being reviewed.

Read from the shared .env or the real environment only, never written into the repo, never requested from a reader on any surface. The repo has never held a credential.

The experimentWe attacked it — the corrected wording, not the document

This reviewer calls decide() on the two-version diff's own primary span, with no retrieval and no ranking in front of it, and unlike docs-route's router the model here never sees the whole document — only the one excerpt diff.primary() selects. So an injection has exactly one door, and a narrower one than docs-route's: the corrected wording, and specifically the piece of it that becomes the largest changed span. Six families were run through that door — a materiality override, a fabricated severity level, forced abstention, a JSON-format break, a system-prompt leak, and a confidence-floor bypass aimed squarely at the same guardrail mechanism docs-route uses: a model's own reported certainty is the only signal the floor has. Measured on 2026-08-07, run x002-docs-redline: 20 document pairs (half the 40-pair corpus) crossed with 6 attack families, 120 live calls on one model, every injection asserted into the span the model was about to be asked to judge before it was paid for. Extends x001 (same day, the first 2 of these 20 pairs, 12 calls).

Four of six families landed on effectively every attempt — override and forced abstention both cleared 20 of 20, the fabricated level and the confidence-floor bypass both cleared 19 of 20. Only the format break (4 of 20) and the system-prompt leak (0 of 20) were meaningfully resisted. The confidence row is the one that matters most for what ships: this kit's own real (non-adversarial) runs already found the floor escalated 1 of 40 pairs on the fast tier and 0 of 40 on the reasoning tier — a floor that almost never fires on its own turns out to be one a document can also just switch off on request, 19 times out of 20.

The resultFour of six attack families got through on 19 or 20 of 20 attempts each, on 20 document pairs and 120 attempts — the confidence-floor bypass that matters most for this kit's guardrail landed 19 times in 20, meaning the check meant to catch an uncertain answer almost never got the chance to fire.

On 20 of the corpus's 40 pairs — half the corpus, the first run at real statistical weight — five of six families got through at least once: materiality override 20/20, forced abstention 20/20, confidence-floor bypass 19/20, fabricated level 19/20, and format break 4/20. Only the system-prompt leak was refused on every attempt. The confidence result is the one worth reading twice: src/materiality.py's CONFIDENCE_FLOOR only ever sees the number the model itself chooses to report, with no independent check on it — and at real sample size, a document that simply instructs the model to always claim 1.0 succeeded on 19 of the 20 pairs it was tried against.

HonestyWhat this does not prove

• 20 of 40 pairs. Half the corpus, and a real sample — but the remaining 20 would still narrow the interval further, especially on the two attacks (format, exfil) with few or no successes.
• One provider, one model, one day. A resistance rate is one vendor's behaviour on one sample, not a property of this system.
• Six attack families, all written by the same person who built the pipeline. An adversary who had not seen this kit's own published finding — that the confidence floor almost never fires on real traffic — would write different, and likely sharper, ones.
• ⚠︎ THE SPAN EACH ATTACK LANDS IN IS CONSTRUCTED, NOT DISCOVERED. decide() only ever judges diff.primary()'s largest changed span, and re-diffing a document with the attack spliced into it in the obvious place scattered the attack text across several small opcodes on most pairs instead of keeping it in the span the model actually reads — word-level diff on two independently-worded texts fragments, which is diff.py's own documented behaviour, not a bug in this harness. So the harness finds the real correction's span on the clean pair, appends the attack to it directly, and hands decide() that exact span rather than re-deriving it. That measures what decide() does once an attack has reached the span with certainty — it does NOT measure how often an attacker could naturally get their payload to become the primary span in the first place, which the redteam_page note below already named as a real, separate, still-open question about src/diff.py's alignment step.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

If the model's own confidence in its materiality verdict is below 0.6, do not publish the verdict — escalate the pair to a person instead. An escalated pair is a correct outcome, not a failure.

src/materiality.py — decide(), CONFIDENCE_FLOOR. Applied in code after the reply is parsed, the same place docs-route's floor lives: nothing downstream sees a verdict the floor withheld, because decide() returns materiality=None, escalated=True before app.py or the run log ever reads it.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• ⚠︎ IT ALMOST NEVER FIRES, AND ONE MODEL NEVER FIRED IT AT ALL. On r001 (the fast tier) the floor escalated exactly 1 of 40 pairs; on r002 (the reasoning tier) it escalated 0 of 40 — every published verdict came from a stated confidence at or above 0.6. A floor that essentially never triggers is not yet load-bearing, it is a number sitting in the code — the same honest finding docs-route's own confidence floor measured.
• It is not a check on whether the verdict is RIGHT. The floor reads the model's stated confidence, which is the model's own opinion of itself. Nothing here says a confidently-stated verdict is a correct one — see Eval.could_not_verify: 'whether either model is actually right on the pairs where they agree' is explicitly not something this kit's agreement design can answer.
• It says nothing about a pair crafted to defeat it — now measured. See security below: the confidence-floor bypass attack succeeded on 19 of 20 adversarial pairs, against the identical mechanism this bullet used to call untested. docs-route's own confidence-floor bypass succeeded on 2 of 20 — a lower rate, but the same guardrail failing the same kind of test.

WatchedWhat is watched, and why that one

The kit measures 30 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

6 measured by the latest run 24 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (pairs 40) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

7 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

materiality · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on confidence_floor, max_tokens, provider, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

compare · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 6 chips that all say so.

DeviationsWhat deviated

0 breaches across 7 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-score the whole set whenever the model, the prompt, src/materiality.py's LEVELS or CONFIDENCE_FLOOR changes — those four are the seams that move the numbers and none of them is comparable across the change. The free regex baseline (evals/baseline.py) can be re-run on every commit for nothing: it calls no provider, and its 40.0%/53.8% agreement against the two models is the cheapest early warning here.

In one lineNo framework, deliberately

This kit has no framework and no dependencies beyond the standard library, the same choice docs-route made and for the same reason: the whole materiality decision is five files you can read in one sitting.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing in this pipeline loops, branches or retries: one call per pair, one verdict, one floor. A graph earns its place when a cycle appears, and align → prompt → model → floor is a straight line with no cycle in it — the same shape docs-route's own graph note describes, for the same reason.

The other sideWhat a framework costs you

• A dependency tree, and a version treadmill on somebody else's release schedule.
• An abstraction over the one thing this kit exists to show you: the exact aligned span the model receives, and the single confidence comparison that decides whether its verdict is published.
• Nothing here approaches the one exception docs-route names — constrained decoding. A materiality reply is a level, a confidence number and a sentence, not a closed set of routing keys; the closest equivalent would be a JSON schema on the reply shape, and 4 of 40 fast-tier replies already fail to produce valid JSON at all — a schema constrains the SHAPE of a reply, not whether the model reasons its way past the token ceiling before writing one.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-docs-redline-flash on the fast tier, 2026-08-07. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 2 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

1 run not plotted. b000-docs-redline-regex recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — a PAIR is the unit; alignment is computed per pair.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: Public domain. Federal Register documents are edicts of government and works of the United States Government, not subject to copyright protection in the United States (17 U.S.C. §105). The Office of the Federal Register places no restriction on reuse of the documents or of the API's output. Verified against federalregister.gov/developers/documentation/api/v1 on 2026-08-06, same verification docs-route already carries for the same API. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineAgreement between two independent model runs

For each pair: did model A and model B land on the same materiality verdict (material / editorial / unsure) for the same aligned span?

MethodHow the test was run

python -m evals.compare --a <run-id> --b <run-id>, offline, no key, no cost.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: itself — agreement has no ground truth to be measured against; it measures two readers, not one reader against an answer key.

How tight can the band be? 34 comparable pairs is the whole denominator this run has — one flipped verdict moves the headline rate by roughly 3 points. No band is published beyond that; a tolerance is the spread across repeated runs, and the prompt and corpus have not been rerun yet to measure one.

Cadence: Re-run both models whenever the prompt or src/diff.py's alignment changes.

The decisionWhen to reach for it