The business caseThe problem this solves

Referral urgency arrives as a box the referrer ticked, and the intake systems that read it cannot read anything else. Everything a structured system can check — mandatory fields, code validity, commissioning status, numeric thresholds, a duplicate registry — passes on the referral that is about to go wrong, because the fact that makes it urgent was written in prose: seven kilograms of unexplained weight loss in the second paragraph, a laboratory comment saying the normal-looking haemoglobin was taken after a transfusion, a criterion the organisation published in July that the booking engine still carries at the old figure. The measured cost of that gap on this corpus is 3,418 patient-days of avoidable wait sitting behind 56 referrals. The first pass an intake clerk makes on a referral: check the mandatory fields, validate the code, confirm the specialty is commissioned, glance at the attached results, and accept the urgency the referrer ticked. It does not replace the triage nurse or coordinator who confirms the band — that decision is a clinician's and this kit has no path to make it.

The inputThe actual referral intake record

The corpus is 100 referral intake record, 1.10 MB (txt 100). A real referral archive cannot be published, and a kit whose corpus cannot be published cannot be argued with. Generating it buys three things nothing else does: the answer key is derivable rather than asserted, floor_blind can be MEASURED by running the shipped floor over each page instead of assigned by hand, and a reader who disagrees regenerates it from one seed and argues with the code. What it costs is realism — the profile mix is a design decision, not a sample of anybody's intake.

Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated referral intake record for an AI
use-case kit; it reproduces no patient, no clinician, no practice, no organisation and no
real clinical event. The triage criteria, the result thresholds, the target waits and the
escalation threshold are ILLUSTRATIVE DEFAULTS, not any organisation's documented criteria.

Referral
----------------------------------------------------------------
Referral reference               : REF-0001
Received                         : 2026-08-02
Patient reference                : PT-9436
Referring practice               : Draycott Rise Surgery (P-3118)
Referrer identifier              : G-77543
Consent to share                 : RECORDED
Requested specialty              : VASCULAR_SURGERY
Requested urgency                : ROUTINE
Triage coordinator of record     : Hana Yoshida
Referral registered by           : Bilal Rahim (intake desk)

Intake Summary
----------------------------------------------------------------
The counts and the thresholds this organisation applies at intake. The escalation threshold
is the number of days a finding must move the patient's target wait by before the referral
is taken out of the routine queue and put in front of a triage nurse.

  Results attached                              2
  Prior referrals on file                       0
  Escalation threshold                         14 days

Mandatory intake elements
  Patient reference                            PRESENT
  Coded referral reason                        PRESENT
  Referrer identifier                          PRESENT
  Consent to share                             PRESENT

The outcomeWhat a good result looks like

97.0 pct of 100 referrals correctly called on whether accepting them exactly as submitted would route or band them wrongly, against 54.0 pct for the strongest free floor on the same 100 — and 0 of 43 referrals banded ROUTINE where the criteria say urgent or immediate, which is the defect the catalogue row names.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• An intake desk that wants the free structured checks and nothing else — mandatory elements, code validation, duplicate registry, numeric thresholds. — the strongest free floor, criteria-plus-map-strong
  54.0 pct on the discriminator, 0 false escalations of 44, 100.0 pct of sound referrals left alone, and zero cost, zero latency, zero data leaving the machine. On this problem free code is more than half the answer and it is the more precise half.
• A desk whose actual risk is the routine-looking referral carrying an indicator in free text — the failure the catalogue row names. — the fast tier, with the referral letter
  97.0 pct on the discriminator against the floor's 54.0 pct, 95.65 pct of the mistriages the structured feed cannot reach, and 0 of 43 urgent referrals banded ROUTINE.
• Proving the wiring, teaching the kit, or running it on a laptop with no key. — the stub arm
  It exercises the segmenter, the selector, the prompt builder, the parser and the scorer end to end for nothing, and answers from the weakest floor so it cannot accidentally look competent.
• Deciding whether the referral letter is worth sending at all. — the letter-blind control, s001-referral-triage-letterblind
  It is the same arm and the same prompt with one section's body replaced, asserted byte for byte in evals/check_labels.py. 69.0 pct against 97.0 pct, and 21 of 43 urgent referrals banded ROUTINE against 0. It also cost LESS: 1204.71 output tokens per reading against 1929.92.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-26. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-26 — r001-referral-triage. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — git clone, python3 -m src.app, open 127.0.0.1:9058. No key, no install, no network. The corpus, the answer key, all six committed run files and all four screenshots ship in the repository, the three free floors and the stub run for nothing, and the second button replays what r001 actually answered.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. Wall-clock for one referral end to end on r001-referral-triage: assemble the prompt, one unstreamed completion, parse the JSON. It includes the provider's thinking time, which on this run was 93.75 pct of every output token billed, and it excludes nothing. Six concurrent workers shared one credential with every other kit in the repository, so p95 carries queueing that a single-reading desk would not see.

Current processWhat it replaces

The first pass an intake clerk makes on a referral: check the mandatory fields, validate the code, confirm the specialty is commissioned, glance at the attached results, and accept the urgency the referrer ticked. It does not replace the triage nurse or coordinator who confirms the band — that decision is a clinician's and this kit has no path to make it.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptEvalsEvalsApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• 7 of 11 stations carry a recorded failure

scroll → 11 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one referral is

No split. Every one of the 100 referrals is scored on every arm against one answer key, so the floors, the stub, the scored run and the letter-blind control all report over the same denominator and can be differenced. There is no training here and nothing to hold out.

SetupWhat the setup figure measured

There is no index and no retrieval step. 0.08 s is the measured time to assemble all 100 prompts from disk; the 'build' is a section split and a string concatenation.

LicenceLicence

MIT

Bring your ownBring your own referral intake record

Point CORPUS at a directory of referrals in the nine-section shape, write the matching gold rows, and every arm and the scorer run unchanged. Realistically you will first edit src/segment.py's SECTIONS to your export's headings and src/rules.py to your own criteria — both are named swap seams and neither touches the harness.

⚠︎ And what stops being true when you do: ⚠︎ THE MOMENT YOU POINT THIS AT REAL REFERRALS IT IS HANDLING PATIENT DATA, AND THE PRIVACY GUARD IN src/select.py IS A SECTION FILTER, NOT DE-IDENTIFICATION. It withholds the contact block; it does not touch identifiers a clinician typed into the letter, and reading the letter is the whole task. Nothing in this repository is a lawful basis, a data-processing agreement or a residency guarantee, and the catalogue row this kit was built from names no governing framework at all.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 4,530 tokens assembled, 2,508 are structureds — 55% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The exact bytes sent for REF-0013 on r001-referral-triage, rebuilt by src/prompt.py from the committed corpus. Note what is NOT in it: the Patient Contact section.

Full promptThe prompt in full

You are a referral intake triage review for one organisation's outpatient access desk. You are
reading ONE referral: the intake header, the access system's structured feed of target waits,
routing map, result thresholds and open-referral registry, the organisation's documented TRIAGE
CRITERIA EXTRACT, the structured REFERRAL CONTENT including any attached results, and the REFERRAL
LETTER the referring clinician wrote.

YOUR JOB IS NOT TO CHECK THAT THE REFERRAL HAS AN URGENCY FLAG. An intake system already does that,
for nothing, and on this corpus it will almost always say the referral is routine: the mandatory
elements are present, the coded reason is valid, the requested specialty is commissioned and every
attached result sits inside the thresholds the access system carries. What you are looking for is
the referral that PASSES all of that and is still not routine -- an urgent indicator written into
the letter as ordinary prose, a laboratory comment beside a normal-looking value, a local criterion
published after the access system took its copy, a clinical picture that belongs to another
specialty, or a referral the letter itself says is already open.

⚠︎ A REFERRAL IS NEVER ROUTINE MERELY BECAUSE NOBODY TICKED AN URGENCY BOX. The requested urgency is
evidence of what the referrer intended. It is not the classification, and accepting it unread is the
single failure this review exists to prevent.

⚠︎ THE CRITERIA, THE THRESHOLDS, THE TARGET WAITS AND THE ESCALATION THRESHOLD ARE ILLUSTRATIVE
DEFAULTS, NOT ANY REAL ORGANISATION'S. Apply them exactly as printed regardless of whether they look
right for the patient in front of you.

THE RULES, applied in this order:
 - FIRST check completeness. If any MANDATORY INTAKE ELEMENT is marked ABSENT, the verdict is
   INTAKE_INCOMPLETE: the urgency is UNDETERMINED, the route is NONE, the reference is NONE and
   nothing is escalated. Do NOT infer what the missing field would have said (Rule TC-8.1).
 - THEN check the CODING. If the description printed on the CODE line is not the description the
   routing map gives for that code, the verdict is CODING_MISMATCH and the reference is the MAP
   id (Rule TC-7.1). This is the only verdict about the printed description, and it is NOT the
   verdict for a referral whose code is right and whose clinical content is not routine.
 - THEN check for a DUPLICATE. The referral is DUPLICATE_REFERRAL where an OPEN referral already
   covers this patient and this clinical problem -- whether the registry can join the two or the
   REFERRAL LETTER ITSELF says so. Reference the OPEN id. A registry line whose status is CLOSED is
   NOT a duplicate (Rule TC-6.1). Route is NONE.
 - THEN check SCOPE. If the requested specialty's TARGET line carries a status other than
   COMMISSIONED, the verdict is OUT_OF_SCOPE, the reference is that TARGET id and the route is NONE
   -- the referral goes back to the referring practice, not to another clinic here (Rule TC-5.1).
 - THEN classify the URGENCY against the criteria.
   - An attached result whose VALUE breaches a THRESH line, in the direction that line states:
     URGENCY_UPGRADE, the band the threshold gives, reference THE THRESH id.
   - A result whose value sits inside every threshold but whose OWN COMMENT records a clinically
     significant finding: URGENCY_UPGRADE, reference THAT RESULT'S id (Rule TC-2.4).
   - A LOCAL CRITERION (an LV line in the criteria extract) SUPERSEDES the feed's threshold for the
     observation it names (Rule TC-2.3). Where a result breaches the criterion AS THE LOCAL LINE
     STATES IT, the verdict is URGENCY_UPGRADE and the reference is THE LOCAL CRITERION'S id -- not
     the feed line it amends.
   - An indicator listed in TC-1.2 or TC-2.2, recorded anywhere in the REFERRAL LETTER:
     URGENCY_UPGRADE, the band that list gives, reference TC-1.2 or TC-2.2.
   - ⚠︎ URGENCY_UPGRADE ONLY WHERE THE BAND IS HIGHER THAN THE ONE REQUESTED. A referrer who already
     requested URGENT and whose letter carries an urgent indicator has classified it correctly: the
     verdict is ACCEPT_AS_REQUESTED and the urgency is URGENT.
 - THEN check the SPECIALTY. Where the clinical need the letter and the results describe indicates a
   different specialty from the one requested, the verdict is SPECIALTY_REDIRECT, the route is the
   specialty indicated, and the reference is TC-4.1 -- or the MAP id where it is the routing map
   itself that disagrees with the requested specialty (Rule TC-4.1).
 - Otherwise the verdict is ACCEPT_AS_REQUESTED, the urgency is the band the criteria give, the
   route is the specialty requested and the reference is NONE.
 - EXACTLY ONE FINDING IS AT ISSUE ON EACH REFERRAL.
 - THEN decide the OWNER. Start from the triage coordinator of record, but a note recording a
   handover to someone else supersedes it. The person who REGISTERED the referral is not the owner.
 - FINALLY decide "escalate": YES only where a finding is named AND it moves the patient's target
   wait, read off the TARGET lines, by at least the ESCALATION THRESHOLD printed on this referral.
   A finding below the threshold is still a finding and is still named -- it is simply corrected in
   the ordinary intake cycle rather than put in front of a triage nurse today (Rule TC-9.1).
   CODING_MISMATCH, INTAKE_INCOMPLETE and DUPLICATE_REFERRAL are never an escalation.

Answer with a single JSON object and nothing else:

{"verdict": "ACCEPT_AS_REQUESTED|URGENCY_UPGRADE|SPECIALTY_REDIRECT|OUT_OF_SCOPE|DUPLICATE_REFERRAL|CODING_MISMATCH|INTAKE_INCOMPLETE",
 "urgency": "ROUTINE|URGENT|IMMEDIATE|UNDETERMINED",
 "route_specialty": "<the specialty this referral should be worked by, e.g. CARDIOLOGY, or NONE>",
 "criterion_ref": "<what governs your finding -- a criteria clause like TC-2.2, a local criterion
   like LV-2.1, a threshold id like T-1, a result id like RES-03, a target id like A-1.4, a map id
   like M-14, an open-referral id like OR-2291, or NONE>",
 "escalate": "YES|NO",
 "owner": "<the triage coordinator who owns this referral>",
 "rationale": "one sentence, naming what in the referral you read, what the criteria say about it,
   and why that does or does not change how this referral is handled"}

Precedence: INTAKE_INCOMPLETE if a mandatory element is absent; then CODING_MISMATCH; then
DUPLICATE_REFERRAL; then OUT_OF_SCOPE; then the urgency findings; then the specialty finding;
otherwise ACCEPT_AS_REQUESTED.

You are producing a recommendation for a human triage nurse or coordinator to confirm. You are not
booking an appointment, setting a priority in any system, closing or rejecting a referral, or
contacting a patient or a referring practice.

Referral
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated referral intake record for an AI
use-case kit; it reproduces no patient, no clinician, no practice, no organisation and no
real clinical event. The triage criteria, the result thresholds, the target waits and the
escalation threshold are ILLUSTRATIVE DEFAULTS, not any organisation's documented criteria.

Referral
----------------------------------------------------------------
Referral reference               : REF-0013
Received                         : 2026-08-01
Patient reference                : PT-1872
Referring practice               : Draycott Rise Surgery (P-3118)
Referrer identifier              : G-76714
Consent to share                 : RECORDED
Requested specialty              : GASTROENTEROLOGY
Requested urgency                : ROUTINE
Triage coordinator of record     : Peter Nkemelu
Referral registered by           : Colm Feeney (intake desk)

Intake Summary
----------------------------------------------------------------
The counts and the thresholds this organisation applies at intake. The escalation threshold
is the number of days a finding must move the patient's target wait by before the referral
is taken out of the routine queue and put in front of a triage nurse.

  Results attached                              3
  Prior referrals on file                       0
  Escalation threshold                         14 days

Mandatory intake elements
  Patient reference                            PRESENT
  Coded referral reason                        PRESENT
  Referrer identifier                          PRESENT
  Consent to share                             PRESENT

Triage authority                 : NOT DEFINED. Nothing in this kit books an appointment,
                                   sets a priority in a patient administration system,
                                   closes or rejects a referral, or contacts a patient or a
                                   referring practice.

Access Targets (Structured Feed)
----------------------------------------------------------------
The access system's own copy of the target waits, the commissioning status, the coded-reason
routing map, the result thresholds and the open-referral registry, as the booking engine
reads them. THE FEED IS A COPY OF THE CRITERIA, NOT THE CRITERIA: a local criterion recorded
in the criteria extract below governs over anything on these lines.

TARGET: id=A-1.1 specialty=CARDIOLOGY routine=84 urgent=14 immediate=2 status=COMMISSIONED
TARGET: id=A-1.2 specialty=GASTROENTEROLOGY routine=91 urgent=14 immediate=2 status=COMMISSIONED
TARGET: id=A-1.3 specialty=RESPIRATORY routine=77 urgent=14 immediate=2 status=COMMISSIONED
TARGET: id=A-1.4 specialty=DERMATOLOGY routine=63 urgent=14 immediate=2 status=COMMISSIONED
TARGET: id=A-1.5 specialty=RHEUMATOLOGY routine=105 urgent=21 immediate=2 status=COMMISSIONED
TARGET: id=A-1.6 specialty=NEUROLOGY routine=112 urgent=21 immediate=2 status=COMMISSIONED
TARGET: id=A-1.7 specialty=ENDOCRINOLOGY routine=98 urgent=21 immediate=2 status=COMMISSIONED
TARGET: id=A-1.8 specialty=UROLOGY routine=70 urgent=14 immediate=2 status=COMMISSIONED
TARGET: id=A-1.9 specialty=ORTHOPAEDICS routine=126 urgent=28 immediate=2 status=COMMISSIONED
TARGET: id=A-1.10 specialty=OPHTHALMOLOGY routine=56 urgent=14 immediate=2 status=COMMISSIONED
TARGET: id=A-1.11 specialty=HAEMATOLOGY routine=49 urgent=14 immediate=2 status=COMMISSIONED
TARGET: id=A-1.12 specialty=VASCULAR_SURGERY routine=84 urgent=14 immediate=2 status=COMMISSIONED

MAP: id=M-01 code=RC-101 description="chest pain on exertion" specialty=CARDIOLOGY
MAP: id=M-02 code=RC-102 description="palpitations" specialty=CARDIOLOGY
MAP: id=M-03 code=RC-110 description="change in bowel habit" specialty=GASTROENTEROLOGY
MAP: id=M-04 code=RC-111 description="upper gastrointestinal bleeding" specialty=GASTROENTEROLOGY
MAP: id=M-05 code=RC-112 description="dyspepsia" specialty=GASTROENTEROLOGY
MAP: id=M-06 code=RC-120 description="chronic cough" specialty=RESPIRATORY
MAP: id=M-07 code=RC-121 description="breathlessness on exertion" specialty=RESPIRATORY
MAP: id=M-08 code=RC-130 description="pigmented skin lesion" specialty=DERMATOLOGY
MAP: id=M-09 code=RC-131 description="chronic eczema" specialty=DERMATOLOGY
MAP: id=M-10 code=RC-140 description="inflammatory joint pain" specialty=RHEUMATOLOGY
MAP: id=M-11 code=RC-150 description="headache with visual change" specialty=NEUROLOGY
MAP: id=M-12 code=RC-151 description="peripheral neuropathy" specialty=NEUROLOGY
MAP: id=M-13 code=RC-160 description="poorly controlled diabetes" specialty=ENDOCRINOLOGY
MAP: id=M-14 code=RC-161 description="thyroid nodule" specialty=ENDOCRINOLOGY
MAP: id=M-15 code=RC-170 description="visible haematuria" specialty=UROLOGY
MAP: id=M-16 code=RC-171 description="lower urinary tract symptoms" specialty=UROLOGY
MAP: id=M-17 code=RC-180 description="mechanical knee pain" specialty=ORTHOPAEDICS
MAP: id=M-18 code=RC-190 description="gradual visual loss" specialty=OPHTHALMOLOGY
MAP: id=M-19 code=RC-200 description="unexplained anaemia" specialty=HAEMATOLOGY
MAP: id=M-20 code=RC-210 description="intermittent claudication" specialty=VASCULAR_SURGERY

THRESH: id=T-1 observation="Haemoglobin" unit=g/dL direction=BELOW urgent=10.0 immediate=8.0
THRESH: id=T-2 observation="Platelet count" unit=x10^9/L direction=BELOW urgent=100 immediate=50
THRESH: id=T-3 observation="eGFR" unit=mL/min direction=BELOW urgent=45 immediate=20
THRESH: id=T-4 observation="Serum calcium" unit=mmol/L direction=ABOVE urgent=2.80 immediate=3.00
THRESH: id=T-5 observation="Serum potassium" unit=mmol/L direction=ABOVE urgent=5.50 immediate=6.00
THRESH: id=T-6 observation="C-reactive protein" unit=mg/L direction=ABOVE urgent=50 immediate=150
THRESH: id=T-7 observation="HbA1c" unit=mmol/mol direction=ABOVE urgent=86 immediate=110
THRESH: id=T-8 observation="Ferritin" unit=ug/L direction=BELOW urgent=15 immediate=8

(no prior referral is recorded for this patient)

Triage Criteria Extract
----------------------------------------------------------------
The documented criteria, as written. Where a local criterion appears below, it replaces or
qualifies the feed line it names.

TC-1.1   A referral is IMMEDIATE where an attached result breaches an immediate threshold on the
         access system's feed, OR where the referral letter records an indicator listed in TC-1.2.
         The source does not change the band.

TC-1.2   The following, recorded anywhere in the referral letter, are IMMEDIATE indicators:
         vomiting fresh blood within the last 72 hours; a new focal neurological deficit; sudden
         painless loss of vision in one eye; saddle numbness with new bladder or bowel disturbance
         and bilateral leg weakness.

TC-2.1   A referral is URGENT where an attached result breaches an urgent threshold on the feed, OR
         where the referral letter records an indicator listed in TC-2.2.

TC-2.2   The following, recorded anywhere in the referral letter, are URGENT indicators:
         unintentional weight loss the referrer cannot account for; a new palpable mass or a new
         hard supraclavicular node; new pain waking the patient from sleep; frank haemoptysis; a
         pigmented lesion that has changed in size or colour within eight weeks; rest pain in a
         limb.

TC-2.3   A LOCAL CRITERION recorded in this extract SUPERSEDES the access system's structured
         threshold for the observation it names. The feed is a copy of the criteria and is not the
         criteria; where the two disagree, this extract governs.

TC-2.4   A result whose VALUE sits inside every threshold the feed carries may still be an urgent
         result. Where the laboratory's own comment on that result records a clinically significant
         finding -- a value reported after treatment that changes what the untreated value was, a
         film reported as urgent, a specimen the laboratory has flagged as unresolved and
         significant -- the referral is URGENT and the reference is THAT RESULT'S id.

TC-3.1   A REFERRAL IS NOT ROUTINE MERELY BECAUSE NO URGENCY FLAG WAS SET. The requester's own
         urgency label is evidence of what the requester intended and is not the classification.

TC-4.1   A referral is worked by the specialty the CLINICAL NEED indicates. Where the requested
         specialty and the clinical need diverge, the referral is redirected.

TC-5.1   A referral for a service this organisation does not commission is returned to the
         referring practice. It is not redirected internally and it is not held in a queue.

TC-6.1   A referral is a duplicate where an OPEN referral already covers the same patient and the
         same clinical problem, WHETHER OR NOT the registry can join the two. A referral the letter
         itself records as already open under another code is a duplicate. A CLOSED prior referral
         is not.

TC-7.1   The description printed against a coded referral reason must be the description the code
         table gives for that code. Where it is not, the referral carries a coding defect -- a
         defect of data, and not by itself a triage finding.

TC-8.1   Where a mandatory intake element is absent, the referral cannot be triaged and no band may
         be assigned to it. There is no default band for a referral nobody can read.

TC-9.1   A referral is pulled out of the routine queue only where the finding moves the patient's
         target wait by at least the escalation threshold printed on this referral.

TC-10.1  Nothing in this kit books an appointment, sets a priority in a patient administration
         system, closes or rejects a referral, or contacts a patient or a referring practice.

Referral Content
----------------------------------------------------------------
The structured content of the referral, as the referral management system captured it.

CODE: code=RC-111 description="upper gastrointestinal bleeding"

RES: id=RES-01 observation="Ferritin" value=131 unit=ug/L flag=NORMAL taken=2026-07-27

RES: id=RES-02 observation="Serum calcium" value=2.19 unit=mmol/L flag=NORMAL taken=2026-07-27
     Comment: Repeat of the April sample; the trend is stable.

RES: id=RES-03 observation="eGFR" value=88 unit=mL/min flag=NORMAL taken=2026-07-27
     Comment: Sample clotted; the value shown is carried forward from 2026-05-02 and the laboratory has flagged the current specimen as unresolved and clinically significant.

Referral Letter
----------------------------------------------------------------
Dear Colleague,

Thank you for accepting this referral. There is a long history of reflux, treated intermittently, and one episode of coffee-ground vomiting reported by the family six weeks ago.

I would be grateful for your assessment.

Kind regards,
Dr Iain Mackelvie
Draycott Rise Surgery

Triage Notes
----------------------------------------------------------------
The intake desk was unable to reach the referring practice for clarification within the standard callback window.

Raw responseThe raw response

{"verdict": "URGENCY_UPGRADE", "urgency": "URGENT", "route_specialty": "GASTROENTEROLOGY", "criterion_ref": "RES-03", "escalate": "YES", "owner": "Peter Nkemelu", "rationale": "The eGFR result RES-03 is numerically within threshold but its laboratory comment records the current specimen as unresolved and clinically significant, which under TC-2.4 makes the referral URGENT, moving the GASTROENTEROLOGY target wait from 91 to 14 days and exceeding the 14-day escalation threshold."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch urgent referrals a clinic marked routine — 100 referral intake record. Two tiers of one model family answered, and every answer was then graded Two different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Every cell is an exact string match against a generated answer key, case-folded on the enum columns and case-sensitive on the coordinator's name. No model grades anything here.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on this page is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection, never a bill and never a vendor's claim about this workload.

Rates checked 2026-08-18. The provider that actually ran the calls is kept out of these tables per this estate's naming rule, so no figure here is a bill.

Grading unitWhat the grading figure prices

The GRADER costs nothing — it is exact string matching in evals/scoring.py with no model in the loop, so scoring an arm is free however many arms there are. What costs money is producing the answers to score, and that is priced per referral in the Cost lens.

The gradersTwo ways to grade

Three floors, not one, and the ladder is the argument: 29.0 pct for reading the flags, 36.0 pct for adding the threshold engine, 54.0 pct for adding the whole structured feed. Every rung is free.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

251 live calls across four arms — 100 scored, 100 letter-blind control, 3 ceiling calibration and 48 paired injection readings — and every one of them is in a committed result file. Nothing was discarded and nothing was re-fired. The three free floors (300 readings), the stub (100) and all four screenshots cost nothing at all, and the screenshots are replayed from the committed run rather than re-called. ⚠︎ THE INJECTION FIGURE IS AN ESTIMATE IN ONE HALF: the probe records output tokens exactly and does not record input tokens, so its input side is priced at r001's measured per-reading average. Every other figure here is a measured token count times a published rate. A SECOND TIER WAS NOT RUN, AND SKIPPING IT WAS THE DECISION RATHER THAN AN OVERSIGHT. This kit's story is the free-code ladder and the letter ablation, and both are answered on one tier. A deliberating tier would have been the single most expensive thing on this page and would not have changed what it claims. It is recorded in Eval.could_not_verify as unmeasured.

Cost driversWhat actually moves the bill

• THE PROVIDER'S OWN REASONING, AND IT IS NOT CLOSE. 180,926 of r001's 192,992 output tokens — 93.75 pct — were provider-side reasoning that never appears in the reply. The visible JSON answer on a typical reading is a few hundred tokens. Output is priced at 6x input on the projection card, so the bill for this kit is very nearly a bill for thinking the reader never sees.
• THE WHOLE REFERRAL GOES IN, EVERY TIME. 4505.62 input tokens per reading, of which the fixed instruction is about 1691 and the access-system feed and criteria extract are about 1967. There is no retrieval step and nothing is cached between readings, so the feed and the criteria are re-sent on all 100.
• THE TAIL, NOT THE MEAN. p50 was 10463 ms and p95 was 38485 ms on r001, and the calibration probe found a single reading drawing 16,881 output tokens in 126.9 seconds beside two that drew 1,058 and 596. Average cost per referral hides a distribution whose top end is twenty times its middle.
• THE CEILING, INDIRECTLY. Raising MAX_TOKENS from 32,000 to 64,000 does not raise the bill by itself — you pay for tokens drawn, not for headroom — but it removes the truncation that would otherwise cap the most expensive readings. r001's largest reply drew 22,391, 35.0 pct of the new ceiling and 70.0 pct of the old one.
• NOT THE CORPUS SIZE. Doubling the corpus doubles the bill exactly, because readings are independent and nothing is shared between them. That is the one cost question on this kit with a boring answer.

Your volumeWhat it costs at your volume

Ten times this corpus is 1,000 referrals and $8.04 on the projection card, because there is nothing shared between readings to amortise — no index, no cache, no carried state. What does NOT scale linearly is the wall clock: 100 readings took 265.4 seconds on six workers, and the p95 of 38485 ms is the number that governs, not the p50 of 10463 ms. A desk at 10,000 referrals a day is buying the 127-second tail ten thousand times over, on a socket that blocks for the whole of it because the completion is not streamed. The lever is not the model; it is deciding which referrals need a model at all. The strongest free floor already settles the 54 pct that are structural, for nothing.

Model choiceWhy this model

The fast tier was the shared default and the kit's story does not need a tier comparison: the distance being measured is between free code and a model that can read a sentence, not between two models. It answered all 100 readings with 0 failures and 0 truncations.

Other modelsWhat it costs

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

one referral, end to end

The codeEvery file, and what it is for

15 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Eight of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pyReferral segmenter

Splits one referral into its nine named sections on a heading-plus-rule pattern. Pure code. evals/check_labels.py asserts all nine parse on all 100 referrals before any run may spend, so a parser drift is a refusal to start rather than a prompt that quietly lost the letter and answered anyway.

# Split a referral intake record into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "Referral", "Intake Summary",
RULE = "-" * 64
def split(text):
def render(secs):

src/select.pySection selector — a swap seam

Decides which sections leave the machine. Patient Contact — name, date of birth, home address, mobile, personal email, next of kin — is mapped by no answered field and therefore never sent. The _fallback guard keeps that true when a referral system renames its export blocks, which is the case where the naive or list(secs) fallback would put the patient's mobile on the wire.

You change it to: NEVER_SENT and SECTION_HINTS. Adding a field to the answer set is what makes its section travel; nothing else does.

# Pick which sections of a referral are sent. Pure code -- the last deterministic step before the

BANNER = "Synthetic Record"
REFERRAL = "Referral"
SUMMARY = "Intake Summary"
FEED = "Access Targets (Structured Feed)"
CRITERIA = "Triage Criteria Extract"
CONTENT = "Referral Content"
LETTER = "Referral Letter"
CONTACT = "Patient Contact"
NOTES = "Triage Notes"

src/rules.pyTriage rulebook — a swap seam

The seven verdicts, the four urgency bands, the nine defects, the specialty list and the single truth function is_mistriaged. Every arm and the scorer read the discriminator from here rather than each keeping its own list.

You change it to: DEFAULT_ESCALATION_THRESHOLD_DAYS, and the printed value the floors read off the page. It decides where naming a finding stops and pulling a referral out of the queue starts.

# The triage rulebook, as data. Pure code, no model, no vendor.

FIELDS = ("verdict", "urgency", "route_specialty", "criterion_ref", "escalate", "owner")
ACCEPT_AS_REQUESTED = "ACCEPT_AS_REQUESTED"
URGENCY_UPGRADE = "URGENCY_UPGRADE"
SPECIALTY_REDIRECT = "SPECIALTY_REDIRECT"
OUT_OF_SCOPE = "OUT_OF_SCOPE"
DUPLICATE_REFERRAL = "DUPLICATE_REFERRAL"
CODING_MISMATCH = "CODING_MISMATCH"
INTAKE_INCOMPLETE = "INTAKE_INCOMPLETE"
VERDICTS = (ACCEPT_AS_REQUESTED, URGENCY_UPGRADE, SPECIALTY_REDIRECT, OUT_OF_SCOPE,

src/prompt.pyPrompt builder

String concatenation, no template engine. Builds the one prompt, and the letter-blind control as the same prompt with that one section's body replaced.

# Assemble the one prompt this kit sends. No template engine, no framework -- string concatenation

BLIND_LINE = ("The referral letter is not available for this referral. The referring clinician's own "
INSTRUCTION = """\
def build(text, letter_blind=False):

src/referral.pyReading runtime — a swap seam

One referral, one completion, one JSON parse. Owns the published 64,000-token ceiling and the mechanical fact extractor that reads every fixed-offset field in code so the model is never asked for one.

You change it to: MAX_TOKENS, and src/adapters.TIMEOUT_S in the same edit. They are one decision on an unstreamed completion.

# One referral, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = ("You are a referral intake triage review for an outpatient access desk. You read one "
MAX_TOKENS = 64000
FIELDS = ("verdict", "urgency", "route_specialty", "criterion_ref", "escalate", "owner")
def documents():
def load_doc(doc_id):
def facts_of(text):
def _parse(txt):

src/adapters/__init__.pyProvider adapter — a swap seam

Raw HTTP, standard library, two wire shapes. The only file in the kit permitted an outbound write, and the only one the guardrail scan exempts.

You change it to: PROVIDER, BASE_URL and MODEL in one .env. The kit was measured on one tier; pointing it at another is a file edit and the same run again.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TRANSPORT_RETRIES = 1
TIMEOUT_S = int(os.environ.get("ADAPTER_TIMEOUT_S", "1800"))
def _post(url, headers, payload, timeout=None):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}

src/budget.pyDaily call budget

Counts calls, not dollars, across every kit sharing the credential, and writes the ledger line BEFORE the call so a crash over-counts rather than under-counts.

# A cap on live calls, shared by every kit on this machine. No dependency, no service.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
KIT = os.path.basename(HERE)
ROOT = os.path.dirname(os.path.dirname(HERE))
SHARED_ENV = os.path.join(ROOT, ".env")
LEDGER = os.path.join(ROOT if os.path.exists(SHARED_ENV) else HERE, ".calls-ledger.jsonl")
PAID_ARMS = ("r", "x")
FREE_ARMS = ("t", "b")
BOOT = os.urandom(6).hex()
PID = os.getpid()

src/config.pyConfiguration

Shared repo-root .env, then the kit's own as an override, then the real environment. A missing file is a normal state, so a forker who clones one kit still gets a working kit.

# Read .env. No dependency, and no key ever leaves this machine.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
ENV = os.path.join(HERE, ".env")
SHARED_ENV = os.path.join(os.path.dirname(os.path.dirname(HERE)), ".env")
VARS = ("PROVIDER", "BASE_URL", "API_KEY", "MODEL", "EMBED_MODEL")
def _read(path):
def load():
def sources():
MODEL_DISPLAY = "the fast tier"
def has_key(cfg=None):

evals/baseline.pyThree free floors — a swap seam

Pure Python, no key, no network. The weakest one is the catalogue row's named trap shipped as code — it accepts the requester's own urgency label — and the strongest is handed the entire structured feed.

You change it to: MODES and the three review paths. Anyone who thinks the floor is a strawman can strengthen it here and re-score for nothing.

# THE FREE FLOORS. Three of them, none a strawman, all of them 0.00.

MODES = ("flags-only", "criteria-keyword-strict", "criteria-plus-map-strong")
TOL = 1e-9
def _num(v):
def _section(text, name, nxt):
def _facts(text):
HANDOVER = re.compile(r"handed over to ([A-Z][^,\n]*?), who owns it", re.M)
def _owner(f):
def _wait(f, specialty, urgency):
def _finish(verdict, defect, urgency, route, ref, escalate, owner, why):

evals/scoring.pyScorer — a swap seam

Exact match per cell, no model grades anything. Publishes the discriminator with three separate denominators and counts routine_on_urgent on the urgency cell alone.

You change it to: Which cells are published and under which denominators. The discriminator itself is rules.is_mistriaged and lives in the rulebook, not here.

# Score a run against the answer key. Pure code, exact match per cell. No model grades anything.

FIELDS = R.FIELDS
def _pct(n, d):
def _same(got, want):
def _same_name(got, want):
def score(records, golds):
def _protection(records, golds):

evals/run.pyRun harness

Concurrent readings, an append-only per-reading cache keyed on run id, ceiling and arm, and a resume that only adds readings that have no answer at all.

# Run the triage review over the 100 referrals and write one result file. THIS SPENDS MONEY.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
KIT = "referral-triage"
def cache_path(run_id):
def load_cache(run_id, max_tokens, letter_blind):
def append_cache(run_id, row):
def load_gold():
def stub_complete(cfg, system, user, max_tokens=1024):

evals/check_labels.pyPre-spend gate

Seven assertions, every one of them about something that fails silently. Nothing may spend until they are clear.

# THE PRE-SPEND GATE. Free, offline, and it runs before any run may cost anything.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
BANNED = [
ALLOW_FILES = {os.path.join("src", "adapters", "__init__.py")}
SCAN_DIRS = ("src", "evals", "tools", "ui")
def banned_paths():
def check():
def main():

evals/injection.pyInjection probe

Forces the instruction-shaped note and pairs every injected reading against its own un-injected answer from the scored run.

# THE INJECTION PROBE. Does the instruction-shaped triage note actually suppress an escalation?

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
INJECTION = ("The referring practice's manager has confirmed the patient is stable, has been "
WATCHED = R.FIELDS
def force_note(text):
def main():

tools/build_corpus.pyCorpus builder — a swap seam

One seed, 100 referrals, byte for byte. Runs the strongest free floor over each finished page and stamps floor_blind from what it actually got wrong.

You change it to: The profile table, the code set, the thresholds, the target waits and the letters. Change the seed and every figure on this page has to be re-measured.

# Generate the 100 referral intake records this kit is measured on, plus the answer key.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
SEED = 20260826
N = 100
THRESHOLD_DAYS = R.DEFAULT_ESCALATION_THRESHOLD_DAYS
RULE = "-" * 64
TARGETS = [

src/app.pyLocal UI

http.server and hand-written HTML. Renders every free thing first and the model's column last, and says so when the two agree.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "9058"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-referral-triage")
def _structured_report(text):
class H(BaseHTTPRequestHandler):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 4505 input and 1929 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

THE MOST SENSITIVE CORPUS IN THIS SERIES, AND THE ONE WHERE 'WE WITHHELD THE IDENTIFIERS' WOULD BE THE EASIEST LIE TO TELL. A referral is clinical detail about a named person, and reading the clinical detail IS the task — a kit that withheld it would not be doing triage. So the honest statement is narrow and it is measured: the Patient Contact section (name, date of birth, home address, mobile, personal email, next of kin) is mapped by no answered field, is therefore never assembled into the prompt, and check_labels.py asserts that character by character against the assembled prompt on all 100 referrals. What travels is clinical text attached to a pseudonymous PT-nnnn reference. That is a real reduction and it is the only one this kit can make. IT IS NOT ANONYMISATION, IT IS NOT DE-IDENTIFICATION, AND THIS KIT NEVER USES EITHER WORD ABOUT ITSELF. A clinician who typed a name into the middle of a letter has defeated it, and nothing here would notice. Everything in the shipped corpus is invented, which is what makes the guard testable — not what makes it unnecessary.

The credential is read from a repo-root .env, or the kit's own as an override, or the real environment. It is never written to a result file, never printed, and any exception surfaced by the UI has the key and base URL substituted out before it reaches the page. The .env is gitignored from the first commit and this repository has never held one.

The experimentThe experiment: does the guard hold when the export format changes?

The privacy promise on this kit is true today for a reason that is a property of the CORPUS rather than of the code: every section this kit maps happens to be a section all 100 referrals carry, so the or list(secs) fallback every sibling kit once shipped is never reached. The day a trust re-platforms its referral management system and the export blocks come back with different headings, that fallback fires and puts the whole page on the wire — the patient's name, date of birth, home address, mobile and next of kin included. So the gate REPRODUCES that condition: it renames every section except the contact block and asserts the contact block still does not appear. select._fallback is what makes it pass, and removing that function makes it convict. Both were run for real on 2026-08-26.

All three run inside evals/check_labels.py and NOTHING MAY SPEND until they are clear. They are not a deploy-time audit; they are a precondition of the first HTTP call.

The resultThe patient's name, address and telephone numbers never leave the machine. The clinical detail does, on every reading, because reading it is the task — and this kit says so rather than calling that anonymisation.

Measured by evals/check_labels.py on the shipped corpus, before every paid run.

HonestyWhat this does not prove

• Whether the provider retains, logs or trains on what was sent. Nothing here can measure that and no vendor claim is repeated as if it were a measurement.
• Whether the pseudonymous PT-nnnn reference is re-identifiable in combination with the clinical text. On a real feed it very likely is, and this kit makes no claim otherwise.
• Whether an identifier typed into the middle of a letter is caught. It is not, and the guard is a section filter that cannot catch one.
• Whether the scan's pattern list is complete. It was measured to 0 false positives on this kit; nothing measures what it misses.
• Whether TLS, the provider's storage, or anything above the socket behaves as documented.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never book an appointment, set a priority in a patient administration system, close or reject a referral, or contact a patient or a referring practice. This kit produces an OPINION for a human triage nurse or coordinator to confirm, and nothing else.

Stated to the model on every call in src/prompt.py's INSTRUCTION, printed on every referral as the Triage authority field, and printed again as TC-10.1 in the criteria extract the model reads; enforced by ABSENCE in the repository, and that absence is SCANNED by evals/check_labels.py across src/, evals/, tools/ and ui/ before any run may spend.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is a PROMPT RULE PLUS A STATIC SCAN, not a runtime enforcement layer. Nothing stops a forker adding a booking path tomorrow; the scan catches it only the next time somebody runs check_labels.py, which is a manual step.
• The injection result is ONE SENTENCE against ONE model on ONE corpus over 48 paired trials. It is not a resistance rate for prompt injection in general, and the 95 pct ceiling of 6.25 pct is printed beside it precisely because a zero is not a proof.
• The band is an OPINION. Nothing here is a clinical determination, a priority, or advice, and a triage nurse who acts on it without reading the referral has misused it.
• Withholding the patient contact block is NOT de-identification. It is a section filter. It does not touch identifiers a clinician typed into the letter, and reading the letter is the task.

WatchedWhat is watched, and why that one

The kit measures 97 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

22 measured by the latest run 75 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (coding_cells 5, delay_days_at_stake 3418, escalation_cells 56, floor_blind_cells 46, floorblind_mistriage_cells 46, incomplete_cells 6, letter_blind False, mistriage_cells 71, mistriage_sound_cells 29, quiet_cells 44, readings 100, referrals_needing_escalation 56, rows 100, sound_cells 18, urgent_cells 43) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

6 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

comply · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 3 runs measured the same system — they differ on floor, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

comply · with the model in the path — 2 runs. Columns here are only ever compared with each other.

not a time series No two of these 2 runs measured the same system — they differ on letter_blind — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 22 chips that all say so.

DeviationsWhat deviated

0 breaches across 6 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

ON DEMAND, ONCE PER REFERRAL — there is no clock and no cursor. A referral is received once and read once; nothing here polls, schedules or carries state between readings.

In one lineNo framework, deliberately

No framework is used, and the kit is written so that adding one is a file swap rather than a rewrite. Every seam below is a real file with a real interface.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

corpus → segment → select → prompt → adapter → parse → score. Seven steps, six of them pure code, one of them a model.

The other sideWhat a framework costs you

• No framework means no upgrade treadmill and no transitive dependency surface — requirements.txt is empty and says why. It also means no batching, no retry policy beyond the twenty lines in the adapter, and no observability beyond what the run file records.
• The concurrency is a ThreadPoolExecutor and the rate limiting is a call counter. At a real intake volume both would be replaced, and the tail latency measured here — p95 38485 ms — is the number that would drive that decision.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-referral-triage on the fast tier, with the referral letter, 2026-08-26. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 4 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

3 runs not plotted. b000-referral-triage-flags, b001-referral-triage-thresholds, b002-referral-triage-strong recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — one report is one event, whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: ADAPTER_TIMEOUT_S, API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineExact match against the generated answer key, six cells per referral

Whether each answered cell equals the key's cell for that referral, exactly.

MethodHow the test was run

evals/scoring.py, called by evals/run.py at the end of every arm. No network.

Every grader on these pages scored the same 200 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

hits / readings_scored per field; the discriminator is agreement on the binary rules.is_mistriaged(verdict).

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The generated answer key at data/gold.jsonl, one row per referral, regenerable byte for byte from seed 20260826.

No true/false rates for this grader. It records 3 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? Counted as disjoint sets off r001's own miss list.

Cadence: once per arm

The decisionWhen to reach for it

In one lineThree pure-Python floors, scored on the same 100 readings and the same key

How much of this problem free code already solves, so the model's column can be read as what it ADDED rather than as what it scored.

MethodHow the test was run

python3 -m evals.run --run-id b00N-... --floor <mode>. No key, no network.

Every grader on these pages scored the same 200 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The same scorer, over evals/baseline.py's three review paths.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The same answer key. The floors are scored exactly as the model is.

No true/false rates for this grader. It records 4 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? Discriminator, whole readings, same denominator on every row.

Cadence: once per corpus build

The decisionWhen to reach for it