The business caseThe problem this solves

Regulatory documents arrive continuously and the cost of misfiling them is asymmetric: a proposed rule filed as a notice silently forfeits the only window in which comment is possible, while a notice escalated to the legal queue merely wastes an hour. A person opening each incoming Federal Register document and deciding which of three queues it belongs in — binding, still open for comment, or for information.

The inputThe actual documents

The corpus is 120 documents, 0.08 MB (txt 120). The gold label is the PUBLISHER'S. The Office of the Federal Register assigns a type to every document it prints, years before this kit existed and with no knowledge of this eval — so there is a right answer nobody here wrote, and == is a legitimate verdict. That is the whole reason this kit can be scored by machine when docs-summarise cannot.

The outcomeWhat a good result looks like

Each document lands in one of three queues within about two seconds, with the queue's consequence stated rather than its label alone.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• High volume, errors cheap to undo — the keyword router
  79.2% for nothing, and every mistake is visible in the queue it lands in
• A taxonomy of your own — flash
  the keyword rules are corpus-specific and would have to be rewritten; the model needs only src/taxonomy.py edited

And where nothing here is good enough:

• Errors expensive and asymmetric — neither, yet
  the guardrail does not work — self-reported confidence is 0.85+ on every wrong flash answer and 0.90+ on every wrong pro one (one at a full 1.0), so nothing here reliably tells you which ~9% to check

At a glanceHow the whole thing runs

Run twice over the same set, for real, the last on 2026-08-07. Every figure on these pages was captured from those runs — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-07 — r003-docs-route-flash. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Clone, python -m tools.fetch_corpus, python -m tools.build_corpus, python -m evals.run --run-id b001 --baseline keyword. Four commands, no key, no cost, and a scored board at the end of it.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. the model call only, one per document, 119 documents run serially (the 120th failed on a network timeout before any reply arrived). It is NOT the wall clock of the run — that was 336 seconds — and it excludes the corpus build, which needs no model and takes under a second.

Current processWhat it replaces

A person opening each incoming Federal Register document and deciding which of three queues it belongs in — binding, still open for comment, or for information.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureEvalsEvalsPromptPromptPromptPromptArchitectureArchitectureApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 9 stations carry a recorded failure

scroll → 9 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one document is

none — a Federal Register abstract is short enough to send whole, so there is no chunker and no packer. This is the first kit here with nothing between the corpus and the prompt.

SetupWhat the setup figure measured

There is no index and no preparation step at all. The corpus boundary does the only work: it decides what the router may see (title, the publisher's one-line action, abstract) and what it may not (the agency, and the type — which is the gold label and lives in a separate file nothing but the scorer opens).

LicenceLicence

Public domain. Federal Register documents are edicts of government and works of the United States Government, not subject to copyright protection in the United States (17 U.S.C. §105). The Office of the Federal Register places no restriction on reuse of the documents or of the API's output; its developer terms ask only for a courteous request rate and no misrepresentation of the data as official. Verified against federalregister.gov/developers/documentation/api/v1 on 2026-08-06. This kit makes three requests in total, one per document type, with a one-second pause and an identifying User-Agent.

Bring your ownBring your own documents

Point tools/build_corpus.py at your own documents and write your own data/gold.jsonl — one line per document, {doc_id, queue}. Then edit src/taxonomy.py to name your queues and what each one commits you to. Nothing else changes: the prompt, the scorer, the baseline and the UI all read the taxonomy rather than holding their own copy of it.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 1,827 characters assembled, 1,388 are instructions — 76% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Rebuilt verbatim by src/prompt.build() over document 2026-15021 — the same function the run called, not a transcription.

Full promptThe prompt in full

You are a routing desk for incoming regulatory documents. You read one document and send it to exactly one queue. You do not summarise it, you do not advise on it, and you do not explain the regulation — a downstream reader does that. Your only job is the queue.

Route this document to one queue.

QUEUES:
  "rule"  (Rule) — BINDING. A final rule is already law or is about to be; the comment window has closed. This queue is the one with a deadline attached — somebody has to read it against what the organisation currently does and say whether anything must change.
  "proposed"  (Proposed Rule) — OPEN FOR COMMENT. Nothing binds yet, and there is a window in which saying something is still possible. Routing a proposed rule to the binding queue wastes a lawyer's morning; routing it to the FYI queue silently forfeits the only chance to influence it, which is the more expensive of the two mistakes and the reason this class exists separately.
  "notice"  (Notice) — FOR INFORMATION. Meetings, filings, applications, statements of policy. The overwhelming majority of what the Federal Register prints, and the queue whose job is to stay boring.
  "unsure" — you are not confident enough to choose. A person will read it.

Answer with JSON only, in this exact shape:
{"queue": "<one key from the list above>", "confidence": <0.0 to 1.0>, "why": "<one short sentence, at most 20 words>"}

DOCUMENT:
TITLE: Evonik Corporation, Filing of Food Additive Petition (Animal Use)
ACTION: Notification of petition.

The Food and Drug Administration (FDA or we) is announcing that we have filed a food additive petition, submitted by Evonik Corporation, proposing that we amend our food additive regulations to provide for the safe use of ethyl cellulose as a binder and coating on amino acids incorporated into food for ruminant animals.

Raw responseThe raw response

{"queue": "notice", "confidence": 0.98, "why": "This is a routine OMB paperwork review notice, informational only."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Sort new federal rules and notices for a compliance team — 120 documents. Two tiers of one model family answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

Deterministic — routed queue == the gold label, which is the Office of the Federal Register's own type field. No judge, no rubric, no person. Every outcome that is not a match is reported in its own column (withheld, off-menu, unparsed) rather than folded into 'wrong', because those have different causes and different fixes.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Run it twiceThe same set, run again

The re-run at a corrected, fixed 1200-token ceiling REVERSES the original ranking. At 400 tokens flash beat pro by 5 points (89.2 vs 84.2) because pro reasons more and lost more replies to the ceiling (12 of 120, against flash's 6). At 1200 tokens, with truncation removed, pro leads flash by 0.9 points (91.7 vs 90.8) — closer to what the larger tier would be expected to do. The 400-token numbers were never a model comparison; they measured which model got hurt less by an undersized ceiling.

accuracy over the balanced set (120 documents; flash scored over 119 after one network failure) — Averaging across the ceiling change would blend a confounded measurement with a corrected one, which is worse than publishing neither. r003 and r004 are the valid pair; r001 and r002 are kept on the record as the ceiling defect that produced them, not as data points to combine with anything.

Grading costWhat it costs

build/facts/models.json in the Foundry repo — the same card every other kit here prices against. Token counts are the provider's own usage block, never an estimate; only the price per token comes from the card.

Rates checked 2026-08-06. The provider that actually ran every model call here is deliberately not named or priced on this page — a withheld-provider rule this standard enforces regardless of whether the rate exists elsewhere in the repo. A dollar figure whose card cannot be named on THIS page is exactly what this standard forbids, so the two cards above stand in for it.

Grading unitWhat the grading figure prices

Free. evals/score.py is arithmetic over a JSONL of labels that arrived with the corpus — it sends nothing, runs in-process and repeats exactly. That is the whole reason this kit publishes scores on the night it was built and docs-summarise, whose grader is a person, does not.

The gradersOne way to grade, and why it is the only one

THE KEYWORD ROUTER IS THE POINT OF THIS KIT. 79.2% overall and 87.2% among the documents it answered, for nothing, in microseconds. Both models now beat it clearly — flash by 11.6 points, pro by 12.5 — once the ceiling that had been confounding the comparison was corrected. The question an AI product manager has to answer is not 'is the model good' but 'is it worth its bill against thirty lines of if-statements', and this kit answers it with a number on the same 120 documents through the same scorer.

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Set limitationsWhat this set cannot show

40 Rule / 40 Proposed Rule / 40 Notice, capped deterministically by document number in tools/build_corpus.py so a rebuild produces the same 120 documents. It took a fetch of 180 to fill: 12 notices carried no abstract and were skipped, and 48 documents fell over the per-class cap.

Balance is what makes the null baseline 33.3% instead of 69%, and it is what makes the 33.3 / 79.2 / 89.2 spread between null, keyword and model readable at all. It also means the per-class precision figures are NOT the ones a real inbox would produce — the natural distribution is roughly 7 notices to 2 rules to 1 proposed rule, measured at 103/34/13 in a 150-document newest-first sample, so the two rarer classes are over-represented here by design.

The corpus costs $0 to build: the Federal Register API is free and needs no key (tools/fetch_corpus.py), and the gold label is the publisher's own type field, not a human annotation (tools/build_corpus.py) — the only spend anywhere in this kit is on the router calls themselves. No natural-distribution run exists. The balanced set answers whether the router can tell the three classes apart; it does not answer what accuracy looks like against a real inbox's roughly 7:2:1 notice:rule:proposed mix (measured 103/34/13 in a 150-document sample) — a weighted re-score against that mix has not been run.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-06. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingScoring is free here, and it is a measured zero

ZERO, AND A MEASURED ZERO RATHER THAN AN UNPRICED ONE. evals/score.py needs no key: it reads the routed queue, reads the gold label, and counts. Scoring all 120 documents costs nothing, so re-running the evaluation costs a forker exactly one routing pass. The two baselines on the board cost nothing for the same reason — b000 and b001 call no provider at all.

Cost driversWhat actually moves the bill

• The queue block is fixed overhead on every call — it is the same ~300 tokens whatever the document, and on a 590-character median document it is the LARGER half of the input.
• Output, not input, is where the two models diverge: pro spent 25,120 output tokens against flash's 17,066 for effectively the same 119-120 decisions, because it reasons more.
• At the corrected 1200-token ceiling the reasoning pass no longer truncates replies for either model — both runs finished with 0 empty responses — so the output-token gap between them is now a genuine reasoning-depth difference, not a ceiling artifact.

Your volumeWhat it costs at your volume

1,200 documents on flash is about $0.13 and roughly 57 minutes of wall clock, serially, projected from r003's measured 336 seconds over 119 documents. Nothing about the shape changes — this kit is cheap enough that cost is not the reason to prefer the keyword router; latency and determinism are.

Model choiceWhy this model

flash was the default the shared .env already pointed at; pro was run because the kit standard requires two models on the score table and one model is a result rather than a trade-off.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The work behind every number on these pages: 120 Federal Register documents, one queue decision each, scored against a set balanced 40/40/40 by construction. Flash's run scored 119 of them — the 120th failed on a network timeout before any reply arrived.

The codeEvery file, and what it is for

Seven modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Four of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/fetch_corpus.pyFetch

three API requests, one per document type, into data/_fetched/

# Pull raw documents from the Federal Register API into data/_fetched/. Costs nothing, no key.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FETCHED = os.path.join(HERE, "data", "_fetched")
API = "https://www.federalregister.gov/api/v1/documents.json"
TYPE_CODE = {"rule": "RULE", "proposed": "PRORULE", "notice": "NOTICE"}
FIELDS = ["document_number", "title", "abstract", "type", "publication_date",
PAUSE_SECONDS = 1.0
def _get(url):
def fetch(per_type):
def main():

tools/build_corpus.pyCorpus boundary

the corpus boundary — writes what the router may see and splits the gold label into data/gold.jsonl

# Turn a raw fetch into the shipped corpus: one .txt per document, a manifest, and the gold set.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FETCHED = os.path.join(HERE, "data", "_fetched")
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
MANIFEST = os.path.join(CORPUS, "manifest.json")
MIN_ABSTRACT = 80
TARGET_PER_CLASS = 40
def _clean(s):
def _document_text(rec):

src/taxonomy.pyThe queues — a swap seam

the three queues and what routing to each one commits you to; every other module reads them from here

You change it to: the queues themselves — change QUEUES and ORDER and the prompt, the scorer and the UI all follow, because none of them holds a second copy of the class list

# The routing taxonomy: the queues a document can land in, and what landing there MEANS.

QUEUES = {
ORDER = ["rule", "proposed", "notice"]
ABSTAIN = "unsure"
FROM_SOURCE = {
def labels():
def label_of(key):
def meaning(key):
def from_source(raw):

src/prompt.pyPrompt and parse

builds the prompt FROM the taxonomy, and parses the reply into four distinct states

# Build the one message pair that asks for a routing decision. Pure code, no model.

SYSTEM = (
def _queue_block():
def build(doc_text):
def parse(text):

src/route.pyThe call, and the floor

the one model call, and the confidence floor that turns a low-confidence answer into an escalation

# The AI layer: one document in, one routing decision out. The whole model surface of this kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 1200
CONFIDENCE_FLOOR = 0.6
def documents():
def load_doc(doc_id):
def decide(cfg, doc_text, complete=None, floor=CONFIDENCE_FLOOR):
def outcome(rec):
def summary_line(doc_id, rec):

evals/baseline.pyThe two free routers — a swap seam

the null router and the keyword router — both free, both on the board

You change it to: the thing the model has to beat. KEYWORDS is thirty lines; a fairer or harsher baseline is an edit to one list

# The two baselines a routing kit has to publish, and neither of them calls a model.

NULL_QUEUE = TX.ORDER[0]
def null_score(n_per_class):
def null_predict(_doc_text):
KEYWORDS = [
def keyword_predict(doc_text):
BASELINES = {

evals/score.pyScorer

confusion matrix, per-class rates, two accuracies, and the floor sweep. The only module that opens gold.jsonl

# Score a set of routing decisions against the gold labels. Pure arithmetic — no model, no judge.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
def load_gold():
def confusion(pairs):
def per_class(m):
def score(records, gold=None):
def floor_sweep(records, gold=None, floors=(0.6, 0.9, 0.95, 0.99, 1.0)):
def sweep_text(sweep):
def board(rows):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 551 input and 143 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

A local, single-user tool over public Federal Register documents. It binds 127.0.0.1, has no auth, no rate limit and no session — correct for a demo on your own machine and wrong for anything else. There is no chat box anywhere in it: the only text a stranger controls is the document being routed.

Read from the shared .env or the real environment only, never written into the repo, never requested from a reader on any surface. The repo has never held a credential.

The experimentWe attacked it — six families, one guardrail

This router calls decide() on the raw document text, with no retrieval and no ranking in front of it — so an injection has exactly one door: the document itself. Six families were run through that door — a routing override, a fabricated queue, forced abstention, a JSON-format break, a system-prompt leak, and a confidence-floor bypass aimed squarely at the guardrail src/route.py already found never fires on its own. Measured on 2026-08-07, run x002-docs-route: 20 documents crossed with 6 attack families, 120 live calls on one model, every injection asserted into the built prompt before it was paid for. Extends x001 (2026-08-06, the first 2 of these 20 documents) rather than replacing a different sample.

Read the confidence row against the other five. Five families got through overwhelmingly — override, forced abstention and format break all landed 18–20 of 20, and only the system-prompt leak was refused outright. The confidence-floor attack is the one that moved with real sample size: 1 of 2 (50%) on the thin run, 2 of 20 (10%) on this one. It still gets through — the guardrail still never fires on its own, src/route.py's own finding stands — but the full run does not support the thin run's suggestion that it fails half the time.

The result5 of 6 attack families got through, most of them overwhelmingly, on 20 documents and 120 attempts — this router had faced no adversarial pressure before x001, and x002 confirms the direction at real sample size rather than a 2-document toe in the water.

On 20 documents — one sixth of the corpus, the first run at real statistical weight — five of six families got through: routing override 20/20, fabricated queue 19/20, forced abstention 19/20, format break 18/20, and the confidence-floor bypass 2/20. Only the system-prompt leak was refused on every attempt. The confidence result is still the one worth reading twice, for a different reason than on the thin run: src/route.py's own real run already found that self-reported confidence never drops below 0.95, right or wrong, so the floor never escalates anything. This attack asked the model to also claim certainty when it should not have, and at real sample size it complied 1 time in 10 — lower than the thin run's 1 in 2, but not zero, and every one of those 2 documents landed a wrong queue with the guardrail never firing.

HonestyWhat this does not prove

• 20 of 120 documents. A real sample at last — six times the thin run — but still one sixth of the corpus, and the confidence-floor rate in particular (2 of 20) has a wide interval at this size.
• One provider, one model, two days. A resistance rate is one vendor's behaviour on one sample, not a property of this system.
• Six attack families, all written by the same person who built the pipeline. An adversary who had not seen the guardrail's own published finding — that confidence never drops below 0.95 — would write different, and likely sharper, ones.
• Nothing was tested against a document long enough to push the real content past the output ceiling, or crafted to defeat something other than the model itself — there is no retrieval or selector here to attack.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

If the model's own confidence in a queue is below 0.6, do not route the document — withhold it for a person. A withheld document is a correct outcome, not a failure.

src/route.py — CONFIDENCE_FLOOR, applied in code after the reply is parsed. It is the only guardrail in the kit, and unlike a prompt rule it is enforcement rather than a request: nothing downstream can see a document the floor withheld.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• ⚠︎ AT ITS DEFAULT SETTING, IT HAS NEVER FIRED — NOW MEASURED, NOT JUST INFERRED. evals/score.py splits withheld into escalated (the floor actually firing) and no_reply (a reply the floor never got to see) as of 2026-08-07, and re-scoring every stored run confirms it directly: r001's 6 withheld and r002's 12 were BOTH 100% no_reply, 0% escalated — the floor did not fire once at 400 tokens either, it just could not be told apart from a ceiling failure before. At the corrected 1200-token ceiling both r003 and r004 read a clean 0.0% on every one of withheld/escalated/no_reply — nothing was withheld for any reason. That confirms the floor still does not fire at the setting shipped; it was never a ceiling artifact hiding the finding, only hiding the proof of it.
• It is not a check on whether the queue is RIGHT. The floor reads the model's stated confidence, which is the model's own opinion of itself; a confidently wrong document passes it untouched. On flash, all 11 wrong answers carried confidence >= 0.85. On pro it is worse: all 10 wrong answers carried confidence >= 0.90, and one was reported at a full 1.0 — no floor below 1.0 would have caught it, and even a floor of exactly 1.0 would not, since 1.0 does not escalate.
• It says nothing about a document that is hostile rather than merely ambiguous. That gap is now measured separately — see security below — and the same guardrail failure shows up there: the confidence-floor bypass attack succeeded on 2 of 20 adversarial documents.

WatchedWhat is watched, and why that one

The kit measures 25 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 18 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

8 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

routing · with the model in the path — 6 runs. Columns here are only ever compared with each other.

not a time series No two of these 6 runs measured the same system — they differ on confidence_floor, documents_attempted, max_tokens, null_baseline_accuracy, provider, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

redteam · with the model in the path — 2 runs. Columns here are only ever compared with each other.

not a time series No two of these 2 runs measured the same system — they differ on attempts, documents — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 8 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-score the whole set whenever the model, the prompt, src/taxonomy.py or CONFIDENCE_FLOOR changes — those four are the seams that move the numbers and none of them is comparable across the change. The keyword baseline can be re-run on every commit for free, and it is the cheapest early warning here: it calls no provider and its 79.2% is a fixed point the model has to beat.

In one lineNo framework, deliberately

This kit has no framework and no dependencies beyond the standard library. That was a choice, and the point of it is that the whole routing decision is four files you can read in one sitting.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing in this pipeline loops, branches or retries: one call per document, one decision, one floor. A graph earns its place when a cycle appears, and this run still gives a reason to want exactly one — re-ask the single document that failed on a network timeout rather than silently dropping it from the scored set. That is a retry, not a graph.

The other sideWhat a framework costs you

• A dependency tree, and a version treadmill on somebody else's release schedule.
• An abstraction over the one thing this kit exists to show you: the prompt the model actually receives, and the single comparison that decides whether its answer is used.
• Constrained decoding is the exception worth naming: it would make an out-of-queue reply impossible. At the corrected 1200-token ceiling there is no longer a class of empty replies for it to save — the one remaining failure mode measured here (a network timeout) happens before any reply exists, so no schema would touch it.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r003-docs-route-flash on the fast tier, 2026-08-07. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 5 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b000-docs-route-null, b001-docs-route-keyword recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the label set is the only structure.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: Public domain. Federal Register documents are edicts of government and works of the United States Government, not subject to copyright protection in the United States (17 U.S.C. §105). The Office of the Federal Register places no restriction on reuse of the documents or of the API's output; its developer terms ask only for a courteous request rate and no misrepresentation of the data as official. Verified against federalregister.gov/developers/documentation/api/v1 on 2026-08-06. This kit makes three requests in total, one per document type, with a one-second pause and an identifying User-Agent. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineGold-label match

Is the queue this router chose the queue the Office of the Federal Register assigned?

MethodHow the test was run

evals/score.py, in-process, no key. Reads the routed queue against data/gold.jsonl, split from the Federal Register API's own type field by tools/build_corpus.py before any router sees the documents — the classes are the publisher's, not authored for this eval. the floor sweep in evals/score.py, printed by every model run, plus reading both tails by hand: all 7 wrong answers carried confidence >= 0.95, and the 6 withheld documents turned out to be empty replies at the output ceiling rather than escalations.

The inputOne real row, seen by every grader

The formulaWhat it computes

routed_to == gold[doc_id], read directly off the record decide() returns. A withheld (escalated) document has routed_to=None, so it can never equal a gold label and is counted as incorrect in `accuracy` — its own column in the confusion matrix, never silently dropped.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: this grader, against gold split from the Federal Register API's own type field by tools/build_corpus.py — not against another model or a human label.

How tight can the band be? There is no threshold to tune — the grader is == and has no knob. What has a denominator worth stating is the balanced set itself: 40 documents per class, so one flipped document moves a class's recall by 2.5 points.

Cadence: Every paid run, and on any change to src/taxonomy.py's QUEUES or tools/build_corpus.py's per-class cap — both change what 'correct' means for at least one document.

The decisionWhen to reach for it