The business caseThe problem this solves

Triaging an adverse-event case means reading the narrative to decide whether the outcome met a regulatory seriousness criterion — a defined list, not an impression — and separately recording the reporter's own causality view, before the case reaches a case processor. The two judgments are easy to conflate, and the report's own severity wording ("severe", "mild") is the most available signal and the wrong one. Someone reading each adverse-event case narrative to decide whether the outcome actually met one of the regulatory seriousness criteria — as opposed to merely sounding bad — and then checking the reporter's own causality assessment before the case is routed to a case processor.

The inputThe actual case reports

The corpus is 55 case reports, 0.03 MB (txt 55). Plain text, one format, invented rather than fetched — a real case narrative cannot be published, and there is no patient or reporter in this one by construction: no names, no dates of birth, no exact ages, only a bucketed age band and a role word. Ten fields are chosen because the same ones drive a real triage decision: which drug, what happened, whether the patient was admitted, the outcome, the reporter's own causality view, and — the field this kit exists to test — whether a regulatory seriousness criterion was actually met.

Case ID
-------
AE-2026-0001

Patient Age Range
-----------------
30-44

Suspect Drug
------------
Orbadanil

Event Description
-----------------
severe swelling of the lips and tongue

Hospitalization
---------------
no

Event Outcome
-------------
recovering

Reporter Causality Assessment
-----------------------------
not-assessed

Reporter Type
-------------
consumer

Case Narrative
--------------
Symptoms were still present at the time of reporting but the patient continues normal daily activity. Nothing beyond stopping the drug has been done and no admission or emergency visit has taken place.

The outcomeWhat a good result looks like

A ten-field extracted record per case, plus one pure-code computed flag folding two checks together — regulatorily serious AND a causality assessment this kit routes on — informational only, never a filed report and never a reporting clock.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Screening an intake queue for which cases a case processor should work first — either tier over the free floor — but read Business.not_good_enough before trusting either flag figure
  96% and 98% seriousness accuracy against the free severity-register floor's 49%, and 23 of 25 against 0 of 25 on the confusable reports the floor is built to fail.
• Deciding the two tiers on cost, speed or accuracy — the fast tier, with a caveat rather than a clean win
  Roughly 11% cheaper per case ($0.0020372 vs $0.0022987), half the p50 latency (3538 ms vs 7084 ms), and 1.00 flag recall against 0.93. But the deliberating tier classified one MORE case correctly (54/55 vs 53/55), so the cheaper tier is not simply the better classifier — it is the one whose errors happened to land where the flag drops them.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-21. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-21 — r001-ae-seriousness. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Verified from a cold clone with no key configured: 55 reports segment into 495 sections in 0.009 seconds, and the assembled prompt for AE-0011 replays byte for byte — its three part sizes match what run r001 recorded, and its measured total of 1313 input tokens matches the capture call's own reported prompt_tokens exactly.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per case

Current processWhat it replaces

Someone reading each adverse-event case narrative to decide whether the outcome actually met one of the regulatory seriousness criteria — as opposed to merely sounding bad — and then checking the reporter's own causality assessment before the case is routed to a case processor.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings; a report with none falls back to one whole-document segment so a span still resolves

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 55 reports cut into 495 sections by src/segment.py, pure code, no model and no key.

LicenceLicence

MIT — this repository's own licence. Every case identifier, drug name and narrative is invented; a real adverse-event narrative cannot be published either — it is among the most identifying text a company holds.

Bring your ownBring your own case reports

Replace data/corpus/*.txt, write data/fields.json, and supply a gold record per case. SECTION_HINTS in src/select.py maps fields to headings and will need editing for a different report layout; when it does not match, selection falls back to the whole document — slower, more expensive, always correct.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 1,313 tokens assembled, 582 are instructions — 44% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix subtraction against the provider's own tokenizer, not estimated from characters — see results/tokens-p001-ae-seriousness.json.

Full promptThe prompt in full

You extract structured fields from an adverse event case report. You return JSON and nothing else.

RULES, in order of importance:
1. If the report does not state a field, return null for it. Do not infer it, do not compute it, and do not use what you know about the world.
2. `is_serious` is a REGULATORY classification -- the seriousness criteria used across pharmacovigilance reporting -- and NOT a description of how bad the event sounds. Answer `yes` when the report describes ANY of the following having actually happened: the patient died; the event was life-threatening; the patient was admitted to hospital, or an existing admission was prolonged, because of the event; the event left a persistent or significant disability or incapacity; a congenital anomaly or birth defect followed exposure; or the event was medically important enough to require an intervention to prevent one of the outcomes just listed. Answer `no` when none of those is described.
3. DO NOT decide `is_serious` from the report's own severity wording. 'Severe', 'moderate' and 'mild' describe how the event felt and are NOT the regulatory test. A severe-sounding event that fully resolved at home with no medical attention is NOT serious. A mild-sounding event that led to a hospital admission IS serious. Read what actually happened to the patient and match it against the list in rule 2. Read the whole narrative before deciding: four of the six criteria are stated only in the narrative and never in a field of their own.
4. `causality_assessment` is a DIFFERENT judgment from `is_serious` and must not be conflated with it. Causality is the reporter's own stated view of whether the drug caused the event; seriousness is about how bad the outcome was. A case can be serious and unrelated at the same time. Report the reporter's own stated causality; do not form your own.
5. `narrative_severity_word` is the colloquial severity word the report itself uses, copied exactly as written. Return null when the report uses no such word -- do not supply one, and do not translate the classification back into a word.
6. Copy values verbatim from the report wherever possible.
7. Use the exact allowed value for a field that lists them.
8. Return every field named in the schema, even when the answer is null.

---

Extract these fields:
- case_id (string) -- the report's own case identifier, verbatim
- patient_age_range (string) -- the bucketed age range as stated (for example "45-54"). Never an exact age and never any other patient identifier -- if the report states a range, copy the range
- suspect_drug (string) -- the drug named as the suspect product
- event_description (string) -- the adverse event as described, copied verbatim
- narrative_severity_word (string) -- the colloquial severity word the report itself uses for the event -- "severe", "moderate", "mild" -- copied exactly as written, or null when the report uses no such word. It is recorded so a reader can see the report's own wording separately from the classification; it must NEVER be used to decide is_serious
- hospitalization (enum) one of: yes, no, unknown -- was the patient admitted to hospital, or was an existing admission prolonged, because of this event
- event_outcome (enum) one of: recovered, recovering, not-recovered, fatal, unknown -- the outcome of the event as stated in the report
- causality_assessment (enum) one of: related, possibly-related, unrelated, not-assessed -- the REPORTER's own stated view of whether the suspect drug caused the event. This is a DIFFERENT judgment from is_serious and must not be conflated with it -- causality is about whether the drug caused the event, seriousness is about how bad the outcome was. A case can be both serious and unrelated. Report the reporter's stated causality; do not form your own
- reporter_type (enum) one of: physician, pharmacist, consumer, other-healthcare-professional -- who reported the case
- is_serious (enum) one of: yes, no -- the REGULATORY seriousness classification, not a description of how bad the event sounds. Answer yes when the report describes ANY of these having actually happened: the patient died; the event was life-threatening; the patient was admitted to hospital or an existing admission was prolonged; the event left a persistent or significant disability or incapacity; a congenital anomaly or birth defect followed exposure; or the event was medically important enough to require an intervention to prevent one of the outcomes just listed. Answer no when none of those is described. Do NOT judge this by the report's own severity wording: a severe-sounding event that resolved at home with no medical attention is NOT serious, and a mild-sounding event that led to a hospital admission IS serious

Return a JSON object with exactly these keys: case_id, patient_age_range, suspect_drug, event_description, narrative_severity_word, hospitalization, event_outcome, causality_assessment, reporter_type, is_serious
Use null for any field the report does not state.

CASE REPORT
-----------
Case ID
-------
AE-2026-0011

Patient Age Range
-----------------
0-17

Suspect Drug
------------
Orbadanil

Event Description
-----------------
mild abdominal pain

Hospitalization
---------------
no

Event Outcome
-------------
recovered

Reporter Causality Assessment
-----------------------------
possibly-related

Reporter Type
-------------
consumer

Case Narrative
--------------
Within twenty minutes of the dose the patient deteriorated to the point that the treating clinician recorded the episode as immediately life-threatening and called for resuscitation support. The patient was stabilised in the treatment room over the following hour. No admission was required.

Raw responseThe raw response

{
  "case_id": "AE-2026-0011",
  "patient_age_range": "0-17",
  "suspect_drug": "Orbadanil",
  "event_description": "mild abdominal pain",
  "narrative_severity_word": "mild",
  "hospitalization": "no",
  "event_outcome": "recovered",
  "causality_assessment": "possibly-related",
  "reporter_type": "consumer",
  "is_serious": "yes"
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch serious drug side-effect reports, even when they sound mild — 55 case reports. Two tiers of one model family answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran r001 and r002 publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersThree ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The severity-register floor (evals/baseline.py) and the scorer (evals/judge.py) are both pure code and cost $0.00 to run against either result set. The figure above is both runs' own token counts (the fast-tier and deliberating-tier runs, 55 reports each) priced at Google Gemini 3 Flash's published rate — the same basis cost_per_query_usd uses, not a second, larger spend.

Cost driversWhat actually moves the bill

• The fixed system prompt and field schema (1117 of 1313 tokens on the example call, 85%) outweigh the case report sent (196 tokens) many times over. Stating all six seriousness criteria in full is what makes it that large — and the free floor that skips them gets 28 of 55 classifications wrong, so this is the floor every call pays to get the judgment right.
• Output length: the model returns a full ten-key JSON record every call regardless of the narrative's own length.

Your volumeWhat it costs at your volume

Linear in cases: each call is independent and self-contained, with no shared context or index to amortise. This run's 55 reports cost about $0.112 projected onto Gemini 3 Flash's rate, so ten times the set is about $1.12 on the same rate and the same prompt — arithmetic on the measured per-call rate, not a second run.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 55 adverse-event case reports extracted, each scored by pure code against a mechanically-derived gold set. This run (r001-ae-seriousness, the fast tier) answered all 55 of 55 reports with no truncation -- the row this table prices, the same one Cost.cost_by_model[0] uses.

The codeEvery file, and what it is for

Five modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Four of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the case report into addressable sections, pure code

# Cut a document into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections carry each field, pure code

You change it to: map fields to your own report's headings; unmatched falls back to the whole document

# Pick which sections plausibly carry each field. Pure code -- the last deterministic step

SECTION_HINTS = {
def for_field(secs, field):
def plan(secs, fields):

src/prompt.pyprompt

assemble one call for all ten fields, with the six seriousness criteria stated in full

# Assemble the extraction prompt. One prompt per case report, all ten fields in it.

SYSTEM = (
def field_schema(fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key — plus the pure-code routing computation downstream

You change it to: this kit's own flat routing rule — replace with your own programme's actual documented criteria before trusting the flag for anything real. It is not a reporting clock and does not model one.

# Extract one case report's fields: segment, select, prompt, one model call, then a pure-code

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 4000
CAUSALITY_TRIGGERS = ("related", "possibly-related")
def load_fields():
def load_doc(stmt_id):
def documents():
def compute(values):

evals/judge.pyjudge

score field accuracy, the flag's recall/precision, and seriousness accuracy by register — three separate figures, pure code

# Score an extraction run. PURE CODE -- gold is exact and the answer is one value per cell, so

def norm(v):
def _num(v):
def equal(field, got, want):
def score(fields, records, golds):
def score_flags(flags, golds):
def score_seriousness(records, golds):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 1303 input and 461 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's reports are entirely synthetic (tools/build_corpus.py) -- no untrusted party wrote any narrative. In a real deployment a case narrative arrives from a physician, a pharmacist, a consumer or a call-centre transcription -- exactly the kind of externally-authored input this kit's architecture reads verbatim and trusts, with no verification step before its text reaches the model. No attack has been tried against this kit; see redteam.why.

Read from the shared .env or the real environment only, never written into the repo, never requested from a reader on any surface. src/app.py's /api/extract handler strips api_key and base_url out of any exception message before it reaches the browser, so a misconfigured key cannot leak into a UI error.

The experimentWe did not attack it — and it failed on ordinary input anyway

An indirect prompt injection needs a field an outside party controls that reaches the prompt. In THIS corpus every report is generated by tools/build_corpus.py, so there is no live untrusted text in the run this report measures -- but a real deployment's narratives arrive from reporters, and that surface has not been attacked. The four gates below are boundaries confirmed by reading the code, not payloads run through it. The fourth is the one to read: it did not need an attacker. Confirmed by reading the code, not by a run, on 2026-08-21 -- no attack run was fired; see redteam.why.

The first three boundaries hold, confirmed by reading the code, not by an attack trial. The fourth does not hold even on ordinary input: is_serious is the field the whole guardrail depends on, it is the one field this kit's span mechanism cannot check because it is an enum rather than a copied value, and three plain reports moved it the wrong way on this run. No crafted input has ever been tried against it.

The result0 attack trials, and three of four boundaries hold in code: no write path exists, the routing rule is a fixed constant the model cannot move, and a misconfigured key cannot leak into a UI error. The fourth failed on ordinary input -- three plain reports moved is_serious the wrong way with no attacker involved.

This run's corpus is entirely generated -- no report's text was authored by an outside party. A real deployment's narratives arrive from reporters and transcriptions, exactly the kind of externally-supplied text this kit reads verbatim and trusts. The number worth reading is the third: before any attacker is involved, three plain reports already talked both tiers out of a correct 'yes' by pairing an intervention with the words 'no admission'. Whether a narrative crafted on purpose could do this reliably is unmeasured.

HonestyWhat this does not prove

• Whether a narrative crafted on purpose could move is_serious reliably -- no red-team run exists for this kit, and the three natural failures this run found suggest the surface is real rather than theoretical.
• Whether the live app's own /api/extract behaves identically to the registered run under an adversarial report -- both use the same src/extract.py::extract(), but neither has been tested against one.
• Whether a code-level consistency check on is_serious (re-deriving the six criteria from the narrative by pure code) would catch a model's wrong answer in practice -- none has been built or exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

A case is routed for expedited review when the model's own is_serious is 'yes' AND its own causality_assessment is one of CAUSALITY_TRIGGERS ('related', 'possibly-related') — computed in pure code from the model's own extracted fields, never asked of the model directly, never overridden by anything it returns.

src/extract.py::compute() -- CAUSALITY_TRIGGERS is a module-level constant read once at import time. Unlike a prompt instruction the model could ignore, this guardrail cannot be talked out of firing by anything in the reply text; it only depends on the model's own is_serious and causality_assessment being right in the first place -- a DIFFERENT, prompt-level dependency the flag inherits, and the one this run found errors in (see is_not).

EvidenceDoes it hold?

The limitWhat a guardrail is not

• IT IS NOT A CHECK ON is_serious ITSELF, AND THIS RUN SHOWED WHY THAT MATTERS. The flag trusts the model's own answer completely; nothing re-derives it from the narrative before computing. Both tiers made classification errors on this corpus -- two on the fast tier, one on the deliberating tier, all three the same criterion -- and the ONLY reason the fast tier still scored 1.00 recall is that both of its errors landed on cases whose causality assessment the flag drops anyway. The deliberating tier's single error did land on a triggering case and became a false negative. A guardrail that inherits an upstream field's accuracy is exactly as good as that field, and on this run that was visible.
• It does not check whether the reporter's stated causality is REASONABLE -- it copies it. A case whose reporter wrongly recorded 'unrelated' is dropped by this rule and nothing here notices.
• It is not a regulatory reporting deadline and does not model one. A real expedited reporting obligation is a clock with a defined start event and a defined day count; CAUSALITY_TRIGGERS is this kit's own flat simplification. See README and data/SOURCES.md.
• It has not been attacked. Whether a narrative crafted to describe a qualifying outcome in routine, self-limiting language could suppress is_serious or needs_review is unmeasured -- see Security.could_not_verify.

WatchedWhat is watched, and why that one

The kit measures 17 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

8 measured by the latest run 9 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 55, extraction_cells 550, failures 0, refusal_cells 7, thinking True) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

2 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 2 runs measured the same system — they differ on the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 2 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

Nothing fires yet. Not one of the 8 bands on this kit names a consequence, so there is no row to print. Which is the honest answer to the heading: a band that nothing is wired to is a number somebody has to notice, and this kit has only those. An empty table would have said the same thing while looking like a rendering failure.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to CAUSALITY_TRIGGERS in src/extract.py, or to the is_serious rule in src/prompt.py -- either changes what the flag is computed FROM. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library -- requirements.txt is empty, with a comment explaining that the emptiness is load-bearing. The whole extraction decision is three files: src/segment.py, src/select.py and src/adapters/__init__.py.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per report -- segment, select, prompt, call, parse, compute -- with no branching and no state carried between reports. A framework would add an orchestrator to a for-loop.

The other sideWhat a framework costs you

• A different field schema (data/fields.json) or a different routing rule needs its own gold set and its own evals/check_labels.py pass -- a framework's own schema layer would not remove that work, only relocate it.
• SECTION_HINTS is a five-minute edit for a new report layout because it is a plain dict, not a configured retriever -- a framework's chunking/retrieval abstraction would need its own re-tuning pass instead, with its own failure modes to learn.
• Swapping providers is one function and one entry in PROVIDERS -- a framework's model abstraction would add a dependency and a version to track for the same one-line change this file already gives away free.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-ae-seriousness on the fast tier, 2026-08-21. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 1 earlier run on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Every case identifier, drug name and narrative is invented; a real adverse-event narrative cannot be published either — it is among the most identifying text a company holds. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-field exact match, light normalisation

Does the model's value for each field match gold, after trimming whitespace and punctuation — and, on the one nullable field, is a correct null scored as a hit rather than a miss?

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every call, and the same field-match logic the free floor is scored by — a baseline and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, derived from the criterion branch that produced each narrative and asserted to appear verbatim in the document; evals/check_labels.py asserts field completeness, enum membership and both states of the nullable field before any run is allowed to spend.

How tight can the band be? There is no tolerance band on the field grade — it is exact match after trimming whitespace and punctuation, never a continuous score to round.

Cadence: Re-score on any change to src/prompt.py, data/fields.json or tools/build_corpus.py — the first two change what is asked, the third changes what is asked about. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineReview-flag confusion matrix

Does the run's own pure-code needs_review (regulatorily serious AND a causality assessment this kit routes on) match the same computation run over gold's own true values?

MethodHow the test was run

evals/judge.py::score_flags, in-process, no key and no model. Runs src/extract.py::compute() over both the run's own extracted values and gold's own values, then compares the two booleans per case.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The same pure-code compute() in src/extract.py, run over gold's own is_serious and causality_assessment values rather than the model's — the TRUE needs_review is never separately typed, only derived.

How tight can the band be? No tolerance band — a binary flag either matches gold's derived flag or it does not, scored per case.

Cadence: Re-score on any change to CAUSALITY_TRIGGERS, src/extract.py::compute, or the prompt's is_serious instruction — all three change what the flag means.

The decisionWhen to reach for it

In one lineSeriousness accuracy, split by the report's own register

On the one judgment field, is the model right BECAUSE it read the outcome, or only because the report's severity word happened to agree? Scored separately on the reports whose severity word points the wrong way.

MethodHow the test was run

evals/judge.py::score_seriousness, in-process, no key and no model. Groups every case's is_serious verdict by the register tag the generator recorded.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: gold's is_serious, together with the register tag tools/build_corpus.py recorded when it generated the report — matching, confusable, or no severity word at all.

How tight can the band be? No tolerance band; each report is right or wrong, counted within its own register.

Cadence: Re-score on any change to AMBIGUOUS_FRACTION or the criterion narratives in tools/build_corpus.py.

The decisionWhen to reach for it