The business caseThe problem this solves

Something produced a set of claims and cited a source for them. The claims read well. Whether the source supports them is a separate question, and answering it by hand means reading both documents in full for every claim. A person reading a summary with the source open beside it, checking claim by claim whether the source actually says what the summary says it does.

The inputThe actual documents

The corpus is 20 documents, 0.02 MB (txt 20). A claim-checking kit needs documents dense with SHORT, SPECIFIC, CHECKABLE assertions. Phase, enrolment, allocation, masking, eligibility bounds, sex, sponsor and the primary outcome's time frame are each a single fact a claim can agree with, contradict, or fail to mention, with no interpretation in between. A narrative document would have forced a judgement call on nearly every row, and a judgement call is not a label. Ten conditions, none overlapping docs-extract's, so the two kits are not reading the same text twice.

The outcomeWhat a good result looks like

174 claims across 20 documents checked in 53.1 seconds on the fast tier, at 0.0190 cents per document. Every verdict comes back with the sentence it rested on, so a reader can overrule it in one glance instead of re-reading the source.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• An unbacked claim must never go out with a tick on it — the reasoning tier (r003)
  0 false supports of 86 unsupported claims — the only run on record with none — against the fast tier's 1, for about 4% more output tokens and roughly 1.4x the latency (p50 3657ms vs 2540ms).
• You are triaging a large backlog and want the hallucinations surfaced cheaply — the fast tier (r002)
  100% recall on not_stated — every genuinely unbacked claim was caught — at $0.000190 a document and a 2.5s p50. The class this kit exists to surface is the class it does not miss.
• You want to know whether you need a model here at all — run the free baseline first (b000-lexical)
  Word overlap alone scores 100% recall on not_stated. If flagging off-topic claims is all you need, that is free and needs no key.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-08. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-08 — r002-docs-verify-flash. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — git clone, then python -m tools.build_corpus — deterministic, standard library only, no network. The raw pulls in data/_fetched/ are never shipped.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. the model call only, one per document, 20 documents run serially. It is NOT the wall clock of the run — that was 53.1 seconds — and there is no retrieval step to include, because this kit resolves a citation rather than searching for one.

Current processWhat it replaces

A person reading a summary with the source open beside it, checking claim by claim whether the source actually says what the summary says it does.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one document is

none — each document is one unit and goes whole into one call; there is nothing to chunk, because these records are nowhere near a context limit and there is no retrieval step to feed

No split — each document is one unit and goes whole into one call. The sizes are the DOCUMENTS themselves (p50 1175, p95 1947 characters), not chunks, because there is nothing to chunk: these records are nowhere near a context limit and there is no retrieval step to feed.

SetupWhat the setup figure measured

There is no index, and this kit is the one where that is a DESIGN DECISION rather than a consequence of short documents. A claim names its source and src/verify.py resolves that name — no embedding, no top-k, nothing to build. The 0.04s is tools/build_corpus.py rendering the 20 documents and deriving their 174 claims, measured as the median of three runs; it calls no model and needs no key.

LicenceLicence

Public domain — works of the U.S. federal government (17 U.S.C. §105). ClinicalTrials.gov is operated by the National Library of Medicine, NIH. No attribution clause, no share-alike, no notice file to propagate. Verified against the registry's terms-of-use page on 2026-08-08.

Bring your ownBring your own documents

Drop .txt files in data/corpus/ and a JSONL of {doc, claims:[{id, text, label}]} at data/claims.jsonl. Nothing else in the kit knows where either came from. If you have claims but no labels, the app and src/verify.py work unchanged — only evals/ needs the labels.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 3,068 characters assembled, 1,950 are datas — 64% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The WHOLE prompt for one document (NCT00289744) on r002-docs-verify-flash — system message then user message, concatenated in the order they were sent. The parts below are literal substrings of it, in ascending order; nothing here retells what was sent.

Full promptThe prompt in full

You check claims against a source document. You are given one document and a numbered list of claims about it. For each claim, decide which of exactly three verdicts applies, using ONLY the document -- never outside knowledge, and never what is likely to be true of studies in general.

  not_stated   The document neither asserts nor denies this. It may be entirely plausible; that is irrelevant. If the document simply does not address it, this is the answer.
  supported    The document states this, or states something that plainly entails it.
  contradicted The document states something incompatible with this.

The distinction between not_stated and contradicted matters more than any other judgement you will make here. A claim the document is silent about is NOT contradicted. Do not reach for a nearby sentence that is merely on the same topic.

For supported and contradicted, quote the sentence or line from the document you relied on, verbatim. For not_stated, the quote must be an empty string -- do not offer the closest line, because a quote that does not decide the claim reads as evidence and is not.

Check each claim below against the document.

Return a JSON object with one key, "verdicts", a list with one entry per claim in the same order, each {"n": <claim number>, "verdict": <one of: supported, contradicted, not_stated>, "quote": <verbatim line from the document, or "">}.

CLAIMS
------
1. This study is a Phase 1 trial.
2. Enrolment was 178 participants.
3. Participants were assigned to groups by a non-randomised design.
4. The trial was run double blind.
5. Participants aged 7 Years to 17 Years were eligible.
6. The study was open to all sexes.
7. The primary outcome was measured over a single visit at week 2.
8. The trial's lead sponsor was GlaxoSmithKline.
9. The results of this trial have been published in a peer-reviewed journal.
10. Participants were compensated for their travel costs.

DOCUMENT
--------
CLINICAL TRIAL RECORD
Registry identifier: NCT00289744

Title: Long-Term Immune Persistence of GSK Biologicals' Combined Hepatitis A & B Vaccine Injected According to a 0,6 Month Schedule

Conditions studied: Hepatitis B, Hepatitis A
Lead sponsor: GlaxoSmithKline
Overall status: COMPLETED
Study start date: 2004-02-16

STUDY DESIGN
Phase: Phase 3
Allocation: NON_RANDOMIZED
Intervention model: SINGLE_GROUP
Primary purpose: PREVENTION
Masking: NONE
Enrollment: 178 [Actual]

ELIGIBILITY
Ages eligible for study: 7 Years to 17 Years
Sexes eligible for study: ALL
Accepts healthy volunteers: Yes

PRIMARY OUTCOME MEASURES
- Anti-hepatitis A Virus (Anti-HAV) Antibody Concentration [Time Frame: Years 6, 7, 8, 9, and 10.]
- Anti-hepatitis B Surface Antigen (Anti-HBs) Antibody Concentration [Time Frame: At Year 6, 7, 8, 9 and 10]
- Anti-hepatitis B Surface Antigen (Anti-HBs) Antibody Concentration [Time Frame: Before and 1 month after the additional dose administration]

BRIEF SUMMARY
The aim of this study is to evaluate the long-term persistence of hepatitis A and B antibodies at Years 6, 7, 8, 9 and 10 after subjects received their first two doses primary vaccination schedule of combined hepatitis A/hepatitis B vaccine. This protocol posting deals with objectives \& outcome measures of the extension phase at year 6 through to 10. The Protocol Posting has been updated in order to comply with the FDA Amendment Act, Sep 2007.

Raw responseThe raw response

```json
{
  "verdicts": [
    {
      "n": 1,
      "verdict": "contradicted",
      "quote": "Phase: Phase 3"
    },
    {
      "n": 2,
      "verdict": "supported",
      "quote": "Enrollment: 178 [Actual]"
    },
    {
      "n": 3,
      "verdict": "supported",
      "quote": "Allocation: NON_RANDOMIZED"
    },
    {
      "n": 4,
      "verdict": "contradicted",
      "quote": "Masking: NONE"
    },
    {
      "n": 5,
      "verdict": "supported",
      "quote": "Ages eligible for study: 7 Years to 17 Years"
    },
    {
      "n": 6,
      "verdict": "supported",
      "quote": "Sexes eligible for study: ALL"
    },
    {
      "n": 7,
      "verdict": "contradicted",
      "quote": "Anti-hepatitis A Virus (Anti-HAV) Antibody Concentration [Time Frame: Years 6, 7, 8, 9, and 10.]"
    },
    {
      "n": 8,
      "verdict": "supported",
      "quote": "Lead sponsor: GlaxoSmithKline"
    },
    {
      "n": 9,
      "verdict": "not_stated",
      "quote": ""
    },
    {
      "n": 10,
      "verdict": "not_stated",
      "quote": ""
    }
  ]
}
```

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check a clinical trial summary's claims against the trial record — 20 documents. Two tiers of one model family answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

Three-way per-claim exact match against labels derived mechanically from the source record. There is NO LLM judge here and that is a choice, not an omission. UC0001 and UC0003 grade with a model because their outputs are free text and no exact rule can decide them. This output is one of three enum values against a label derived from the same structured field the document was rendered from — a model asked to grade that would add cost, latency and its own error rate to a comparison == already decides correctly.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Run it twiceThe same set, run again

These are two MODELS, not two runs of one, so nothing here establishes how much of the 1.72-point gap is noise. There is direct evidence it is not zero: one borderline claim returned SUPPORTED on r002 and CONTRADICTED from a later live call with the same model, same prompt and same document.

accuracy over all 174 labelled claims —

Grading costWhat it costs

Every dollar published on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-01. The provider that actually ran r001/r002/r003 is not priced here; its rate is not committed anywhere in this repo, and a dollar figure whose card cannot be cited is not a figure this site publishes.

Grading unitWhat the grading figure prices

Grading is free: evals/judge.py makes no call and needs no key, so re-scoring any number of runs over 174 claims costs a forker nothing beyond the checking calls themselves. The billable unit of the CHECKING is the document — one call covers every claim about it — which is why 174 claims cost 20 calls.

The gradersOne way to grade, and why it is the only one

The baseline is the most informative row on this page. Word overlap alone gets 100% recall on not_stated — a claim about peer review shares almost no vocabulary with a study record — and then collapses to 21.6% recall on supported, because supported and contradicted differ by ONE value in an otherwise identical sentence. That is a precise statement of what the model is buying: not the ability to notice an off-topic claim, but the ability to compare two values.

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-01. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingGrading is free here, and it is a measured zero

evals/judge.py makes no call and needs no key: grading 174 claims across any number of runs costs a forker nothing beyond the checking calls themselves.

Cost driversWhat actually moves the bill

• The document dominates the prompt — 46.8%% of assembled characters against the claims' 16.8%% — so cost tracks document length, not claim count.
• One call per document, so cost scales linearly with corpus size (see cost_at_10x).
• Every verdict carries a verbatim quote, which is most of the output bill: 290 output tokens per document across ~8.7 claims.

Your volumeWhat it costs at your volume

Linear in documents: 200 documents on the fast tier is about $0.04. It stops being linear when a single document carries more claims than one reply can quote for — then the batch splits and cost returns to scaling with claims.

Model choiceWhy this model

The fast tier is the default because it gets 98.28% for 0.0190 cents a document. The reasoning tier is perfect on this corpus for 3.8% more output tokens — worth it wherever a false 'supported' is expensive, which is the whole reason someone runs a claim checker.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 20 documents checked, 174 labelled claims graded by pure code. The fast tier's run (r002) answered all 174 — the row this table prices, the same one Cost.cost_by_model[0] uses.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Four of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

data/claims.jsonlClaim set

174 claims across 20 documents, each naming the source it is about.

# 

src/verify.pyCitation resolver — a swap seam

Turns a claim's named source into that document's text. Pure code — no index, no retrieval, no top-k.

You change it to: load_claims() reads a JSONL file. Point it at your own extractor's output and the kit checks that instead.

# Check one document's claims: resolve the citation, one model call, check the quotes, done.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
CLAIMS = os.path.join(HERE, "data", "claims.jsonl")
MAX_TOKENS = 2000
def load_doc(doc_id):
def documents():
def load_claims():
def claims_for(doc_id):
def _quote_is_real(quote, doc_text):

src/prompt.pyPrompt assembly — a swap seam

One document plus all of its claims, numbered, in a single message. Pure code.

You change it to: VERDICTS plus the definitions in SYSTEM. A fourth verdict (partially supported) is a two-line change here and a column in evals/judge.py's matrix.

# Assemble the one prompt this kit sends, and parse the one reply it gets back.

VERDICTS = ("supported", "contradicted", "not_stated")
VERDICT_MEANINGS = {
SYSTEM = (
def build(doc_text, claims):
def parse(raw, n_claims):

src/adapters/__init__.pyOne model call — a swap seam

One provider, one key, one call per document.

You change it to: openai-compatible and anthropic both implemented; PROVIDER in .env picks one.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/verify.pyQuote check — a swap seam

Is the sentence the model quoted actually in the document? Pure code, free, and it catches a model inventing its own evidence.

You change it to: load_claims() reads a JSONL file. Point it at your own extractor's output and the kit checks that instead.

# Check one document's claims: resolve the citation, one model call, check the quotes, done.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
CLAIMS = os.path.join(HERE, "data", "claims.jsonl")
MAX_TOKENS = 2000
def load_doc(doc_id):
def documents():
def load_claims():
def claims_for(doc_id):
def _quote_is_real(quote, doc_text):

evals/judge.pyScorer — a swap seam

The 3x3 confusion matrix, per-class recall and precision, false-support rate. Pure code — no judge model.

You change it to: Pure-code exact match today. Swapping in an LLM judge means replacing score() alone; nothing upstream knows how grading happens.

# Score a run against the labelled claim set. PURE CODE — there is no LLM judge in this kit.

LABELS = ("supported", "contradicted", "not_stated")
def _pct(n, d):
def score(records, labelled):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 733 input and 290 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

A local, single-user tool over public ClinicalTrials.gov records. It binds 127.0.0.1, has no auth, no rate limit and no session — correct for a demo on your own machine and wrong for anything else. There is no chat box in it: the only text a stranger controls is the cited SOURCE being checked, which is exactly the surface this run attacks.

Read from the shared .env or the real environment only, never written into the repo, never requested from a reader on any surface. The repo has never held a credential. The UI screenshots are taken with the key blanked, and the shoot script now refuses to start if anything is already listening on its port — a stale copy of this kit holding a real key passes an identity check perfectly, and that gap cost a real provider call elsewhere.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. Reproduce with python -m evals.redteam --run-id x001-docs-verify --docs 5. Measured on 2026-08-08, run x001-docs-verify: 5 documents crossed with 6 attack families, 30 live calls on the fast tier with reasoning disabled — the configuration the app and both scored evals ship. All 30 attempts scored; none returned an empty reply. Each injection was asserted into the document about to be sent before it was paid for, and two free control stubs ran first — one that resists everything (0/6) and one that complies with everything (6/6) — so every detector is known to move in both directions.

⚠︎ THE FORGED-QUOTE ROW IS MEASURED AGAINST THE ORIGINAL DOCUMENT, NOT THE ONE THAT WAS SENT, AND THE REASON IS THE MOST USEFUL THING THIS RUN FOUND. The obvious detector asks whether the pipeline's own quote check failed. It cannot work here: the injection is appended to the DOCUMENT, so the wording the attacker tells the model to cite becomes genuinely part of the text the check compares against — it returns TRUE and the attack reads as resisted. This kit's one code-enforced guardrail proves a quote exists in the document it was GIVEN; it cannot know the line was put there by whoever wrote the instruction. An attacker who supplies their own evidence defeats a substring check completely. docs-comply is immune by construction — its injection lands in the rulebook while quotes are checked against the document — so the same check has opposite outcomes decided entirely by where the attacker can write. Caught by the free control stub before any money was spent.

The result3 of 30 scored attempts followed the attacker — 90.0% resisted, the STRONGEST of the four kits measured. ⚠︎ THIS PAGE PREVIOUSLY CONCLUDED THAT "THE RULEBOOK IS ABOUT FOUR TIMES SOFTER THAN THE DOCUMENT", FROM THIS RUN AND docs-comply's ALONE. Two further runs — docs-redact at 50.0% resisted and docs-summarise at 42.9%, both DOCUMENT-injection kits — show that framing does not hold. The correction is below and the original claim is not quietly removed, because it was published.

The four-kit picture is the finding, and it is not the one two runs suggested. docs-verify — injected into the document, closed verdicts — is this claim supported?: 3 of 30 followed, 90.0% resisted · docs-comply — injected into the rulebook, closed verdicts — does this rule pass?: 13 of 29 followed, 55.2% resisted · docs-redact — injected into the document, span extraction — find every identifier: 15 of 30 followed, 50.0% resisted · docs-summarise — injected into the document, free-text generation — write a brief: 16 of 28 followed, 42.9% resisted. Three of the four sit between 43% and 55%, and the outlier that RESISTS is a document-injection kit — so "rulebook versus document" does not explain the spread. What lines up is what the model is asked to DO with the text. Asked to JUDGE it against a fixed, closed vocabulary — is this claim supported by that source? — the model treats the document as EVIDENCE, and an instruction inside it is just more evidence; it resists at 90%. Asked to ACT on it — summarise it, extract from it, apply rules to it — it treats what it reads as part of the JOB DESCRIPTION and obeys about half the time. docs-comply fits: closed vocabulary like docs-verify, but its injection arrives in the RULES, which are the job description by definition. Four runs, one provider, one day, six attacks each — a pattern across four comparable measurements, not a study.

HonestyWhat this does not prove

• The earlier conclusion on this page was wrong and is corrected above. Two runs suggested the injection's LOCATION explained the gap; four runs show it does not. A comparison built from two points is a line through two points.
• Whether a real attacker would use these six. They were written by the kit's author against the kit's own design.
• Whether the reasoning tier resists differently. x001 ran the fast tier only, and the two tiers already disagree measurably on ordinary judgement.
• The app's HTTP surface. x001 drives verify() directly, the same code path the app calls, but the app was not attacked through its own interface.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Use ONLY the document. A claim the document is silent about is `not_stated`, never `contradicted` — do not reach for a nearby sentence that is merely on the same topic. For `supported` and `contradicted`, quote the line you relied on verbatim; for `not_stated` the quote must be empty.

src/prompt.py — SYSTEM, built from VERDICT_MEANINGS rather than restated beside it. Prompt-level for the verdict itself, but UNLIKE every sibling kit's prompt-only rule, half of it IS enforced in code: verify.py checks the quote is a real whitespace-normalised substring of the document, so a model that invents its evidence is caught by pure code at no cost.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• ⚠︎ IT DOES NOT MAKE A BORDERLINE VERDICT STABLE. The claim 'double blind' against a document reading 'Masking: TRIPLE' returned SUPPORTED on r002 and CONTRADICTED from a later live call — same model, same prompt, same document. The rule tells the model what to decide, not how firmly, and nothing in code checks that two calls agree.
• It is not a defence against a hostile document. Nothing here has been attacked: a source carrying instructions, or a claim written to argue with the checker, is untested. See the security gap in Eval.could_not_verify.
• The quote check proves a sentence EXISTS in the document, not that it DECIDES the claim. A model could quote a real but irrelevant line and pass this check with a wrong verdict.

WatchedWhat is watched, and why that one

The kit measures 21 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 14 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

verify · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on max_tokens, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 7 chips that all say so.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-score both models whenever src/prompt.py changes — VERDICT_MEANINGS is the one declaration the model, the parser, the scorer and the site's app panel all read, so a wording change there changes what is being asked of every one of them. Same on any change to MAX_TOKENS or to the claim set.

In one lineNo framework, deliberately

This kit has no framework and no dependencies beyond the standard library — requirements.txt names nothing, on purpose. The whole checking decision is three files: prompt.py, verify.py, adapters/__init__.py.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing in this pipeline loops, branches or retries beyond the adapter's own transient-error backoff: one call per document, one set of verdicts, pure code the rest of the way. A graph earns its place when a cycle appears, and resolve -> prompt -> call -> parse -> quote-check is a straight line with no cycle in it.

The other sideWhat a framework costs you

• A dependency tree, and a version treadmill on somebody else's release schedule — for a kit whose entire promise is a ten-minute clone.
• An abstraction over the one thing this kit exists to show: the three verdicts, their definitions, and the instruction that a silent document means not_stated. That is roughly 1,100 characters of plain text a reader can hold in their head.
• The closest real exception is constrained decoding — see the mapping above. It is the one framework whose absence here is a judgement call rather than an easy no.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r002-docs-verify-flash on the fast tier, 2026-08-08. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 2 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: Public domain — works of the U.S. federal government (17 U.S.C. §105). ClinicalTrials.gov is operated by the National Library of Medicine, NIH. No attribution clause, no share-alike, no notice file to propagate. Verified against the registry's terms-of-use page on 2026-08-08. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThree-way verdict exact match, plus a quote-presence check

Does the returned verdict equal the label? Separately: is the quoted sentence actually a substring of the document, whitespace-normalised? See evals/judge.py.

MethodHow the test was run

evals/judge.py, in-process, no key and no model. Gold verdicts come from data/claims.jsonl, which tools/build_corpus.py derives from the same structured field the document text is rendered from and refuses to emit if a label cannot be verified against its own document.

The inputOne real row, seen by every grader

The formulaWhat it computes

Per claim: `returned_verdict == label`, over the three values in src/prompt.py's VERDICTS. Nothing is normalised because nothing is free text — parse() has already rejected anything outside the vocabulary, and a claim it returned no verdict for stays None rather than defaulting to a class. Accuracy is reported over ANSWERED claims with coverage beside it, never blended, because one number mixing 'got it wrong' with 'never replied' is what lets a half-broken run look respectable. Separately and for free: the quoted sentence is checked as a whitespace-normalised substring of the document, which is the cheapest hallucination check in the kit.

IterationsFour rulers, four wrong numbers

The tell was the ranking inverting, not the value moving. Noise moves a number; an artifact reorders the comparison the page exists to make.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: this grader, against labels DERIVED from the same structured field the document is rendered from — never against another model's output. The derivation is what makes the labels trustworthy and also what bounds them: they are exactly as good as the templates that produced them.

How tight can the band be? 174 labelled claims is the whole denominator on record; one flipped claim moves accuracy by about 0.57 points, so the 1.72-point gap between the two tiers is three claims. No band exists — r002 and r003 are different models, not two runs of one, and one borderline claim is already known to flip between calls. See Eval.repeat.

Cadence: Re-run on any change to src/prompt.py — VERDICT_MEANINGS is the vocabulary the model, the parser, the scorer and the site's app panel all read, so a wording change there changes what is being asked — or to MAX_TOKENS in src/verify.py, which bounds how many claims can be answered with a quote each.

The decisionWhen to reach for it