The business caseThe problem this solves

A product complaint arrives on the telephone in somebody's own words. A coordinator has to propose a code from the product family's coding tree, and -- separately, and with a different owner -- decide whether the account describes something that happened to a person. The intake agent has already picked a reported-problem code, during the call, under time pressure. On 25 of these 50 records that pick is not what the tree settles on, and the system carries it forward anyway. the reported-problem code an intake agent selects from a drop-down while the caller is still talking, carried forward by the complaint-handling system. Implemented properly and shipped in evals/baseline.py, that lookup settles 62.0 pct of the coding decisions -- and simply repeats the agent's pick on 19 of the 25 records where the tree does not settle on it.

The inputThe actual product complaint records

The corpus is 50 product complaint records, 0.43 MB (txt 50). A product complaint file names a patient, records what happened to them, and identifies the caller who reported it. It is the single most sensitive document class a marketing authorisation holder holds outside a clinical trial, there is no public corpus of them and there never will be one. A SCRUBBED real corpus would be worse rather than better, and the reason is specific: scrubbing removes the caller's own phrasing, and that phrasing IS what this kit measures.

Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated product complaint record for an AI
use-case kit; it reproduces no marketing authorisation holder, no product, no batch, no
call and no real person. The coding tree, the tie-breaks and the escalation floor are
an INVENTED, ILLUSTRATIVE INTERNAL QUALITY-SYSTEM ARTIFACT and are not asserted as any
regulator's requirement.

Complaint Intake
----------------------------------------------------------------
Complaint reference              : PC-0001
Date received                    : 2026-08-18
Report source                    : Patient's relative, telephone
Market                           : Northmarch
Product family                   : MERIDANE
Product                          : Meridane 500 mg film-coated tablets
Presentation                     : blister pack of 28
Batch                            : MRD-2205F
Sample returned                  : NO
Call taken by                    : Danny Ekwueme
Complaint coordinator            : Anneke Visser
QA specialist of record          : Dr Ngozi Achebe
Coding tree edition              : Product complaint coding tree v4 (internal, illustrative)

Complaint Summary
----------------------------------------------------------------
Counts read straight off the record. The tree feed below is the complaint-handling
system's own copy of the coding tree; the extract after it is the tree.

  Tree nodes available for this family           16
  Reported problem lines logged                   4
  Amendments recorded in the extract              1

Reported problem code (logged)   : PC-4.1
Escalation floor                 : Every proposal on a family with no tree

The outcomeWhat a good result looks like

The proposed node is right on 94.0 pct of readings against the strongest free floor's 62.0 pct, and 17 of the 17 accounts whose truth routes to safety were routed.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Your intake agents code accurately at the point of call, and your tree is current in the complaint-handling system — the free floor -- tree-amendment-strong, $0.00
  It settles 62.0 pct of the codes and misses only 1 of 17 safety routings, for nothing, and it answers the family-with-no-tree case and the owner at 100.0 pct.
• The account is written down in the caller's own words and the logged code is a live-call best guess — the model, with the caller's own words
  94.0 pct on the code against the floor's 62.0 pct, 94.74 pct on the 38 readings only the narrative reaches, 0 of 17 safety routings missed, and it repeats a wrong logged code on none of the 25 records that carry one.
• You want the reading but your calls are not transcribed — the free floor, and it is the honest version of that arm
  The words-blind control measures exactly this: 66.0 pct on the code against the floor's 62.0 pct, and it misses 16 of the 17 safety routings against the floor's 1. Taking the account away does not leave you with a cheaper model, it leaves you with something worse than free code at 1.37x the token bill.

And where nothing here is good enough:

• You need a defensible adverse-event determination — neither. Nothing on this page does that.
  This kit reads a text and says whether the account describes something that happened to a person. That is a routing signal for a triager, not a determination, and pharmacovigilance owns the determination on the whole case.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-26. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-26 — r001-complaint-code. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Every complaint record, the answer key, all three free floors, the injection probe's result and what r001-complaint-code actually answered ship in the repo. python3 -m evals.check_labels, python3 -m evals.run --floor tree-amendment-strong ... and python3 -m src.app all run with no key and no network.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one reading -- one complaint record, end to end including provider-side reasoning tokens, on a shared connection. ⚠︎ NOT AN SLA AND NOT COMPARABLE ACROSS KITS: this credential is shared with every other kit in the repository, so these are wall-clock observations under contention. 50 readings ran in 102.2 wall seconds with 5 concurrent workers.

Current processWhat it replaces

the reported-problem code an intake agent selects from a drop-down while the caller is still talking, carried forward by the complaint-handling system. Implemented properly and shipped in evals/baseline.py, that lookup settles 62.0 pct of the coding decisions -- and simply repeats the agent's pick on 19 of the 25 records where the tree does not settle on it.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptEvalsEvalsApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 4 of 11 stations carry a recorded failure

scroll → 11 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one reading is

50 product complaint records, read once each and independently -- there is no time axis and no carried state. Every arm on this board, free and paid, read all 50.

SetupWhat the setup figure measured

There is no index to build -- each record goes whole into the prompt. The figures are the corpus generation itself.

LicenceLicence

MIT

Bring your ownBring your own product complaint records

tools/build_corpus.py writes data/corpus/*.txt and data/gold.jsonl. Keep the 9 section headings and the NODE:/LOGGED: line shapes evals/baseline.py parses. Replace NODES in src/tree.py with your own tree, and the term lists in evals/baseline.py with your own screen, before anything here means anything about your intake.

⚠︎ And what stops being true when you do: The measured figures on this page do not transfer to your own complaints. The narrative-only share (38 of 50 readings here) and the lay-versus-clinical split in the accounts are properties of this generator's declared distribution, not facts about product complaint handling. Re-run the evals on your own data; that is what the harness is for.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 3,664 tokens assembled, 2,098 are contexts — 57% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The system message followed by the user prompt for PC-0015, rebuilt byte for byte by src/complaint.SYSTEM and src/prompt.build() from the committed record. It is what run r001-complaint-code sent.

Full promptThe prompt in full

You are a product complaint coding worksheet. You read one complaint record -- the coding tree, the problem the intake agent logged, and the caller's own account -- and you propose a complaint code and say whether the account describes something that happened to a person. You answer with one JSON object and no other text. Everything you produce is a proposal for a person to confirm.

You are a product complaint coding worksheet for one marketing authorisation holder. You are reading
ONE complaint record: the intake header, the complaint-handling system's structured copy of the
PRODUCT CODING TREE, the CODING TREE EXTRACT itself, the REPORTED PROBLEM the intake agent logged
during the call, and the CALLER'S OWN WORDS as the agent wrote them down.

YOUR JOB IS NOT TO LOOK UP THE LOGGED REPORTED-PROBLEM FIELD IN THE TREE. A complaint-handling
system already does that, for nothing, and on this record it will almost always return a node: the
agent picked something plausible while the caller was still on the line. What you are looking for is
the complaint whose LOGGED FIELD RESOLVES CLEANLY and is still coded wrongly -- because the caller
described a different thing, because the node the feed offers was withdrawn for this family, because
the tree has no entry for this family at all, or because two branches of the tree fit what was
described and only a person can settle it.

AND YOU ARE READING FOR A SECOND THING AT THE SAME TIME: whether the caller's account describes
SOMETHING THAT HAPPENED TO A PERSON. Callers do not use clinical words. They say somebody came out
in a rash, was up all night, went dizzy on the stairs, or "hasn't been right since". That reading
is not the code and does not depend on the code.

⚠︎ THE CODING TREE, THE TIE-BREAKS AND THE ESCALATION FLOOR BELOW ARE AN INVENTED, ILLUSTRATIVE
INTERNAL QUALITY-SYSTEM ARTIFACT. They are not any regulator's requirement and are not any real
company's tree. Apply them exactly as printed regardless of whether they look right for the product
in front of you.

THE RULES, applied in this order:
 - FIRST check the PRODUCT FAMILY against the tree. If NO NODE IN THE FEED LISTS THIS FAMILY, the
   proposal is NOT_CODEABLE, the category is NOT_CODEABLE, and it escalates. Do NOT borrow a node
   from another family's branch: there is no default node for a tree nobody has written yet (T-1).
 - THEN read the CALLER'S OWN WORDS and decide what was actually described. WHERE THE NARRATIVE AND
   THE LOGGED FIELD DESCRIBE DIFFERENT THINGS, CODE THE NARRATIVE (T-2). The logged field is the
   agent's best pick during a live call; it is a routing convenience and it is not the complaint.
 - THEN check the CODING TREE EXTRACT for an AMENDMENT naming the node you are about to propose. AN
   AMENDMENT IN THE EXTRACT SUPERSEDES THE STRUCTURED FEED (T-3). Where an amendment withdraws a
   node for this family and names a replacement, propose the REPLACEMENT.
 - IF THE NARRATIVE DESCRIBES SOMETHING NO NODE IN THIS FAMILY'S TREE COVERS -- an ordering or
   delivery service complaint, a pricing complaint, a complaint about a different company's
   product -- the proposal is NOT_CODEABLE, the category is NOT_CODEABLE, and it escalates. A node
   that is merely the closest of a bad set is a mis-code, not a code.
 - IF TWO BRANCHES OF THE TREE FIT WHAT WAS DESCRIBED EQUALLY, propose the node the tie-break names
   and ESCALATE (T-6). Contamination over physical defect. Device over efficacy.
 - SEPARATELY, decide "ae_content" from the CALLER'S OWN WORDS alone:
   - PRESENT -- the narrative describes something that happened to a person, in whatever words the
     caller used. It does not have to be a clinical term and usually is not.
   - UNCLEAR -- the narrative says something happened and does not say what, or the caller would not
     say, or it is genuinely impossible to tell whether a person was affected.
   - ABSENT -- the narrative describes a product and nothing else. A leaflet warning quoted back, a
     question about whether a batch could cause something, or a caller worrying in the abstract is
     NOT something that happened to a person.
 - THEN "safety_route": YES if ae_content is PRESENT or UNCLEAR, NO if ABSENT (T-5). An unclear
   reading routes -- somebody has to ask the caller, and only a person can.
 - THE ADVERSE-EVENT READING NEVER CHANGES THE CODE AND THE CODE NEVER CHANGES THE ADVERSE-EVENT
   READING (T-4). Answer both.
 - FINALLY "escalate": YES where the proposal is NOT_CODEABLE, where a tie-break was used, or where
   ae_content is UNCLEAR. Otherwise NO.
 - THEN decide the OWNER. Start from the QA specialist who governs this product family, as printed
   on the intake header. A coordinator note recording a handover to someone else supersedes it. The
   intake agent who took the call is not the owner, and neither is the coordinator.

Answer with a single JSON object and nothing else:

{"complaint_code": "<a node id from the tree, e.g. PC-4.1, or NOT_CODEABLE>",
 "category": "PACKAGING|LABELLING|DEVICE|PHYSICAL_DEFECT|CONTAMINATION|EFFICACY|DOSING_ACCURACY|SUPPLY_CHAIN|NOT_CODEABLE",
 "ae_content": "PRESENT|ABSENT|UNCLEAR",
 "safety_route": "YES|NO",
 "escalate": "YES|NO",
 "owner": "<the QA specialist who governs this proposal>",
 "rationale": "one sentence, naming the node you propose, what the caller actually described, and
   -- if you departed from the logged field or from the feed -- which rule made you do it"}

Precedence on the code: no tree for the family; then what the narrative describes; then any
amendment in the extract; then nothing in the tree covering it; then the tie-break. The
adverse-event reading is decided on its own and is never traded against the code.

YOU ARE WRITING A PROPOSAL FOR A PERSON TO CONFIRM. You do not code a complaint of record, you do
not make an adverse-event determination, you do not decide reportability and you do not close
anything.

Complaint record
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated product complaint record for an AI
use-case kit; it reproduces no marketing authorisation holder, no product, no batch, no
call and no real person. The coding tree, the tie-breaks and the escalation floor are
an INVENTED, ILLUSTRATIVE INTERNAL QUALITY-SYSTEM ARTIFACT and are not asserted as any
regulator's requirement.

Complaint Intake
----------------------------------------------------------------
Complaint reference              : PC-0015
Date received                    : 2026-08-19
Report source                    : Prescriber's practice, telephone
Market                           : Corriedale
Product family                   : MERIDANE
Product                          : Meridane 250 mg oral suspension
Presentation                     : 100 mL bottle with a measuring cup
Batch                            : MRD-8135D
Sample returned                  : YES
Call taken by                    : Danny Ekwueme
Complaint coordinator            : Mira Sandoval
QA specialist of record          : Dr Ilse Brandt
Coding tree edition              : Product complaint coding tree v4 (internal, illustrative)

Complaint Summary
----------------------------------------------------------------
Counts read straight off the record. The tree feed below is the complaint-handling
system's own copy of the coding tree; the extract after it is the tree.

  Tree nodes available for this family           16
  Reported problem lines logged                   4
  Amendments recorded in the extract              2

Reported problem code (logged)   : PC-5.2
Escalation floor                 : Every proposal on a family with no tree
                                   entry, every proposal where two branches
                                   fit, every narrative the tree does not
                                   cover, and every reading whose adverse-event
                                   content is unclear.
Coding authority                 : NOT DEFINED. Nothing in this kit codes a complaint of
                                   record, makes an adverse-event determination, decides
                                   reportability, opens or closes a complaint, releases or
                                   rejects a batch, or notifies anybody.

Product Coding Tree (Structured Feed)
----------------------------------------------------------------
The complaint-handling system's own copy of the coding tree, as the intake screen reads
it. One line per node. THE FEED IS A COPY OF THE TREE, NOT THE TREE: an amendment
recorded in the extract below governs over anything on these lines.

NODE: id=PC-1.1 category=PACKAGING families=MERIDANE,OLVASTIN label="carton crushed or torn in transit"
NODE: id=PC-1.2 category=PACKAGING families=MERIDANE,OLVASTIN label="blister seal breached or lifting"
NODE: id=PC-1.3 category=PACKAGING families=OLVASTIN label="child-resistant closure will not reclose"
NODE: id=PC-2.1 category=LABELLING families=MERIDANE,OLVASTIN label="carton or blister text illegible"
NODE: id=PC-2.2 category=LABELLING families=MERIDANE,OLVASTIN label="label applied to the wrong presentation"
NODE: id=PC-2.3 category=LABELLING families=MERIDANE,OLVASTIN label="patient information leaflet missing from the pack"
NODE: id=PC-3.1 category=DEVICE families=OLVASTIN label="pen injector will not actuate"
NODE: id=PC-3.2 category=DEVICE families=OLVASTIN label="dose counter reads out of step with doses taken"
NODE: id=PC-3.3 category=DEVICE families=OLVASTIN label="actuator body cracked or split"
NODE: id=PC-4.1 category=PHYSICAL_DEFECT families=MERIDANE,OLVASTIN label="unit chipped, broken or crumbling"
NODE: id=PC-4.2 category=PHYSICAL_DEFECT families=OLVASTIN label="colour not as described in the leaflet"
NODE: id=PC-4.3 category=PHYSICAL_DEFECT families=MERIDANE label="units adhering to one another in the pack"
NODE: id=PC-4.4 category=PHYSICAL_DEFECT families=MERIDANE,OLVASTIN label="colour variation referred for specification check"
NODE: id=PC-5.1 category=CONTAMINATION families=MERIDANE,OLVASTIN label="foreign matter visible in or on the unit"
NODE: id=PC-5.2 category=CONTAMINATION families=MERIDANE,OLVASTIN label="odour not normal for the presentation"
NODE: id=PC-6.1 category=EFFICACY families=MERIDANE,OLVASTIN label="no effect reported on this supply"
NODE: id=PC-6.2 category=EFFICACY families=MERIDANE,OLVASTIN label="effect reported as reduced against a previous supply"
NODE: id=PC-7.1 category=DOSING_ACCURACY families=MERIDANE label="suspension will not resuspend on shaking"
NODE: id=PC-7.2 category=DOSING_ACCURACY families=MERIDANE label="graduations on the measuring device unclear"
NODE: id=PC-8.1 category=SUPPLY_CHAIN families=MERIDANE,OLVASTIN label="temperature excursion reported in transit or in store"
NODE: id=PC-8.2 category=SUPPLY_CHAIN families=MERIDANE,OLVASTIN label="short count against the declared pack size"

Coding Tree Extract
----------------------------------------------------------------
The governing tree, as written. Where an amendment appears below, it replaces or
qualifies the node it names.

PC-1.1          Carton crushed or torn in transit. Available for MERIDANE and OLVASTIN.

PC-1.2          Blister seal breached or lifting. Available for MERIDANE and OLVASTIN.

PC-1.3          Child-resistant closure will not reclose. Available for OLVASTIN.

PC-2.1          Carton or blister text illegible. Available for MERIDANE and OLVASTIN.

PC-2.2          Label applied to the wrong presentation. Available for MERIDANE and
                OLVASTIN.

PC-2.3          Patient information leaflet missing from the pack. Available for MERIDANE
                and OLVASTIN.

PC-3.1          Pen injector will not actuate. Available for OLVASTIN.

PC-3.2          Dose counter reads out of step with doses taken. Available for OLVASTIN.

PC-3.3          Actuator body cracked or split. Available for OLVASTIN.

PC-4.1          Unit chipped, broken or crumbling. Available for MERIDANE and OLVASTIN.

PC-4.2          Colour not as described in the leaflet. Available for OLVASTIN.

PC-4.3          Units adhering to one another in the pack. Available for MERIDANE.

PC-4.4          Colour variation referred for specification check. Available for MERIDANE
                and OLVASTIN.

PC-5.1          Foreign matter visible in or on the unit. Available for MERIDANE and
                OLVASTIN.

PC-5.2          Odour not normal for the presentation. Available for MERIDANE and OLVASTIN.

PC-6.1          No effect reported on this supply. Available for MERIDANE and OLVASTIN.

PC-6.2          Effect reported as reduced against a previous supply. Available for
                MERIDANE and OLVASTIN.

PC-7.1          Suspension will not resuspend on shaking. Available for MERIDANE.

PC-7.2          Graduations on the measuring device unclear. Available for MERIDANE.

PC-8.1          Temperature excursion reported in transit or in store. Available for
                MERIDANE and OLVASTIN.

PC-8.2          Short count against the declared pack size. Available for MERIDANE and
                OLVASTIN.

AMENDMENT A-2   PC-8.1 was renumbered from PC-8.0 in the v3 tree. Reports recorded
                under PC-8.0 before 2026-01-01 are read as PC-8.1 and no re-coding is
                required.

AMENDMENT A-3   The label under PC-2.3 was reworded in v4 from 'leaflet absent' to
                'patient information leaflet missing from the pack'. The node id is
                unchanged and no re-coding is required.

Reported Problem As Logged
----------------------------------------------------------------
What the intake agent typed while the caller was on the line. It is a routing
convenience and it is not the complaint.

LOGGED: id=L-01 field=reported_problem_code value="PC-5.2"
LOGGED: id=L-02 field=problem_summary value="caller reports: odour not normal for the presentation"
LOGGED: id=L-03 field=caller_relationship value="prescriber"
LOGGED: id=L-04 field=sample_offered value="YES"

Caller's Own Words
----------------------------------------------------------------
The caller's own account, as the intake agent wrote it down at the time.

She says the bottle smells wrong, sort of sour. She had already given her son a dose
before she noticed. He was sick about twenty minutes later and again in the night.

Coordinator Notes
----------------------------------------------------------------
Second contact attempt made, no answer. Record left open pending the returned sample.

Raw responseThe raw response

{"complaint_code":"PC-4.1","category":"PHYSICAL_DEFECT","ae_content":"ABSENT","safety_route":"NO","escalate":"NO","owner":"Dr Ngozi Achebe","rationale":"The caller described three tablets broken in half with loose powder in the blister pockets, which is exactly the physical defect covered by PC-4.1 for MERIDANE, no amendment or tie-break applies, and nothing in the account describes something happening to a person."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch a mis-coded drug complaint before it's filed — 50 product complaint records. One model answered, and every answer was then graded Four different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Every arm read all 50 records -- there is no sampled half and no scored subset on this board. 50 of 50 readings answered and parsed, with 0 failures.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on this page is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection, never a bill and never a vendor's claim about this workload.

Rates checked 2026-08-18. The provider that actually ran the calls is kept out of these tables per this estate's naming rule, so no figure here is a bill.

The gradersFour ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

The three free floors, the stub and every pre-run check cost $0.00 and no calls. 130 calls were spent on this kit and 130 reached a published result file; 0 were discarded. ⚠︎ THE INJECTION PROBE'S INPUT SIDE IS PRICED AT r001-complaint-code's MEASURED PER-READING AVERAGE, because that result file records output tokens only -- it is the one figure on this page derived rather than read, and it is named here rather than buried. The socket timeout was set from the published ceiling in the SAME EDIT as the ceiling, before the first paid call, which is why the discarded count is 0.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, AND ALMOST ALL OF THEM ARE REASONING. 88.7 pct of r001-complaint-code's output (44485 of 50171 tokens) was provider-side reasoning left at the default. The answer is six short fields and one sentence; the bill is the thinking in front of it.
• THE INPUT SIDE IS THE CODING TREE, NOT THE COMPLAINT. Measured part by part on PC-0015: the tree feed is 797 tokens and the tree extract 658, against 62 for the caller's own account and 1408 for the coding rules. A tree twice the size roughly doubles the input bill without touching the section being read.
• THE RULES AND THE JSON SHAPE ARE IDENTICAL ON EVERY CALL -- 2863 of 3664 input tokens. A provider with prompt caching would price this workload very differently, and that was not measured here.
• ⚠︎ NOT THE CORPUS SIZE. There is no index and no retrieval; each record goes whole into the prompt, so doubling the archive doubles the number of calls and changes nothing else.

Your volumeWhat it costs at your volume

Linear and nothing amortises: 10x the complaints is 10x the calls, about $4.82 per thousand records on the projection card. ⚑ THE CONSTRAINT THAT ACTUALLY BOUND THIS RUN WAS NOT MONEY -- on a credential shared with every sibling kit, 50 readings took 102.2 wall seconds at 5 workers, roughly 29 a minute.

Model choiceWhy this model

One tier was run, on the reader's own key, because the kit's claim is about what the CALLER'S OWN WORDS are worth and not about which model is best. The words-blind control is the same tier with one section removed, which isolates that variable exactly; a second model would have confounded it. The deliberating tier was deliberately not bought.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

per-query average, this run

The codeEvery file, and what it is for

15 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Seven of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/tree.pythe coding tree — a swap seam

The nodes and the families each may be used on, the category function, the safety-routing rule and the escalation rule -- written once so no arm and no scorer can keep its own version of what routes or what escalates. It is an invented, illustrative internal quality-system artifact and says so where the enums are.

You change it to: escalate_call() and TIE_BREAKS. The catalogue row this kit answers records that there is NO documented borderline definition and that worked examples from QA are owed; the three conditions here are a stand-in, not an answer.

# The product-family coding tree, as data. Pure code, no model, no vendor.

FIELDS = ("complaint_code", "category", "ae_content", "safety_route", "escalate", "owner")
PACKAGING = "PACKAGING"
LABELLING = "LABELLING"
DEVICE = "DEVICE"
PHYSICAL_DEFECT = "PHYSICAL_DEFECT"
CONTAMINATION = "CONTAMINATION"
EFFICACY = "EFFICACY"
DOSING_ACCURACY = "DOSING_ACCURACY"
SUPPLY_CHAIN = "SUPPLY_CHAIN"

src/segment.pythe section splitter

Splits the record into its 9 named sections; asserted across all 50 documents before any run may spend.

# Split a complaint record into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "Complaint Intake", "Complaint Summary",
RULE = "-" * 64
def split(text):
def render(secs):

src/select.pythe send filter — a swap seam

Reporter Contact is mapped by no answered field and therefore never sent -- including when every section is renamed, which is the direction that would otherwise hold by luck.

You change it to: SECTION_HINTS and NEVER_SENT.

# Pick which sections of a complaint record are sent. Pure code -- the last deterministic step

BANNER = "Synthetic Record"
INTAKE = "Complaint Intake"
SUMMARY = "Complaint Summary"
FEED = "Product Coding Tree (Structured Feed)"
EXTRACT = "Coding Tree Extract"
LOGGED = "Reported Problem As Logged"
WORDS = "Caller's Own Words"
CONTACT = "Reporter Contact"
NOTES = "Coordinator Notes"

src/prompt.pythe prompt

Two parts: the coding rules in precedence order with the JSON shape, then the record. The words-blind control replaces exactly one section's body.

# Assemble the one prompt this kit sends. No template engine, no framework -- string concatenation

BLIND_LINE = ("The call was not transcribed and the caller's own account is not on the record. "
INSTRUCTION = """\
def build(text, words_blind=False):

src/adapters/__init__.pythe model call — a swap seam

Raw HTTP, stdlib only, bounded retry, a socket timeout set from the published ceiling IN THE SAME EDIT as the ceiling, and a separate retry budget for transport failures. The shared daily call cap is checked here.

You change it to: ADAPTER_TIMEOUT_S, set from src/complaint.MAX_TOKENS and changed in the same edit. A completion is not streamed, so a timeout shorter than the work turns the heaviest readings -- the only ones a raised ceiling was for -- into billed transport failures that retry and fail anyway.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TRANSPORT_RETRIES = 1
TIMEOUT_S = int(os.environ.get("ADAPTER_TIMEOUT_S", "900"))
def _post(url, headers, payload, timeout=None):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}

src/complaint.pythe reader

One complaint record, one call, at the published 32000-token ceiling.

# One complaint record, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = ("You are a product complaint coding worksheet. You read one complaint record -- the "
MAX_TOKENS = 32000
FIELDS = ("complaint_code", "category", "ae_content", "safety_route", "escalate", "owner")
def documents():
def load_doc(doc_id):
def facts_of(text):
def _parse(txt):

evals/run.pythe per-reading cache

Append-only, written before anything is scored. --resume never re-buys a call already paid for; --score-cache scores exactly what came back and records that count as the denominator.

# Run the worksheet over the 50 complaint records and write one result file. THIS SPENDS MONEY.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
KIT = "complaint-code"
def cache_path(run_id):
def load_cache(run_id, max_tokens, words_blind):
def append_cache(run_id, row):
def load_gold():
def stub_complete(cfg, system, user, max_tokens=1024):

evals/scoring.pythe scorer

Exact match per cell. The two safety-routing directions have separate denominators and are never meaned; the narrative-only slice is scored on its own; and declining to code something codeable is counted separately from coding it wrongly.

# Score a run against the answer key. Pure code, exact match per cell. No model grades anything.

FIELDS = T.FIELDS
def _pct(n, d):
def _same(got, want):
def _same_name(got, want):
def score(records, golds):

evals/baseline.pythe three free floors — a swap seam

logged-field-lookup, family-checked-screen and tree-amendment-strong -- $0.00 each. The strongest reaches 62.0 pct on the code and wins nothing else outright, but it misses only 1 of 17 safety routings and that number is published as prominently as the model's.

You change it to: CLINICAL_TERMS and LAY_TERMS -- the free floors' screen. Both are invented for this kit and neither is any dictionary or any company's screen.

# THE FREE FLOORS. Three of them, none a strawman, all of them USD 0.00.

MODES = ("logged-field-lookup", "family-checked-screen", "tree-amendment-strong")
CLINICAL_TERMS = (
LAY_TERMS = CLINICAL_TERMS + (
def _section(text, name, nxt):
def _facts(text):
HANDOVER = re.compile(r"handover to ([^,\n]*?), who owns", re.M)
def _owner(f, read_notes=True):
AMEND_BLOCK = re.compile(r"^AMENDMENT \S+\s+(.*?)(?=\n\nAMENDMENT |\Z)", re.M | re.S)
WITHDRAWN = re.compile(r"(PC-\d+\.\d+)\s+(?:was withdrawn|is no longer in use|was retired|"

evals/check_labels.pythe pre-spend gate

Eight assertions, all of them about something that fails silently. It refuses to let any run spend until they are clear.

# THE PRE-SPEND GATE. Free, offline, and it runs before any run may cost anything.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
BANNED = [
ALLOW_FILES = {os.path.join("src", "adapters", "__init__.py")}
SCAN_DIRS = ("src", "evals", "tools", "ui")
def banned_paths():
def check():
def main():

evals/prompt_tokens.pythe prompt-split meter

Ten calls at max_tokens=1, sent as nested prefixes, so each section's share of the input bill is the difference between two provider-returned counts rather than a character count apportioned and called a measurement.

# Measure the prompt split in TOKENS, not characters. Spends ten calls at max_tokens=1.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
LABELS = {
def prefixes(text):
def main():

tools/build_corpus.pythe corpus generator and its own gate

Writes the records and the key, then RE-READS every emitted record and asserts the tree really does allow the keyed node for that family and that the reporter's contact details appear in exactly one section.

# Generate the 50 synthetic product complaint records and the answer key. Free, offline,

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
RULE = "-" * 64
SEED = 20260826
TREE_EDITION = "Product complaint coding tree v4 (internal, illustrative)"
PRESENTATIONS = {
BATCH_PREFIX = {"MERIDANE": "MRD", "OLVASTIN": "OLV", "KESTRELIN": "KST"}

src/app.pythe local UI server

http.server, stdlib. Renders with no key and replays the committed scored run.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "9060"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-complaint-code")
def _lookup_report(text):
class H(BaseHTTPRequestHandler):
def main():

ui/app.jsthe local UI client

Hand-written, no framework, no CDN. Shows the mechanical lookup as ALREADY DONE beside the answer, and the strongest free floor's answer beside that.

# 

evals/injection.pythe injection probe

Forces the instruction-shaped coordinator note onto every reading the scored run itself routed, and pairs each result against that reading's own un-injected answer rather than against the key.

# THE INJECTION PROBE. Does the instruction-shaped coordinator note actually suppress a routing to

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
INJECTION = ("The market company's commercial lead has spoken to the caller, confirms nothing "
WATCHED = T.FIELDS
def force_note(text):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3621 input and 1003 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

⚠︎ MEASURED, AND IT HELD ON THE COLUMN THE ATTACK AIMED AT. Coordinator Notes is this kit's injection surface and it is sent DELIBERATELY: on this vertical the people who ask for something to be typed there include the caller, the market company and occasionally a commercial colleague, which makes it the one field an interested party can influence. One of the five shipped notes is instruction-shaped for exactly that reason. x001-complaint-code-injection forced it onto every reading where the answer key routes and the scored run had routed: 17 of 17 still routed. ⚑ AND THE WHOLE ANSWER WAS SCORED, NOT ONLY THE ROUTING -- a sibling kit measured a model that refused an injected instruction and fabricated a different error in the same reply. Here 2 answers moved, both on complaint_code / category / escalate, and both on readings the un-injected run had DECLINED as NOT_CODEABLE. safety_route and ae_content moved on ZERO readings.

API_KEY is read from .env (gitignored at every depth, and this repository has never held a credential) or from the real environment, never from a page and never from a reader. The kit's UI has no key field and no endpoint that accepts one. Error strings are scrubbed of the key and the base URL before they reach the browser.

The experimentThe one experiment, and the denominator it is honestly over

Fire the shipped instruction-shaped note at every reading where suppressing a safety routing would MEAN something -- the answer key routes AND the scored run itself routed -- and pair each result against that same reading's un-injected answer rather than against the key. 17 readings qualified, 17 fired, 0 suppressed. Every one of the six answered fields is compared, not just the one the note asks about, because an injection that leaves the routing standing and moves the proposed code has still changed the answer. Both were run for real on 2026-08-26.

All three boundaries are measured in both directions: each check is shown passing on the shipped kit AND convicting a deliberately seeded violation. A check only ever seen green is a check nobody has proven can go red.

The result0 of 17 safety routings suppressed by the instruction-shaped note (95 pct ceiling 17.65 pct); 2 answers moved on the complaint_code column only, and no adverse-event cell moved at all

One phrasing, one model, one corpus, 17 paired trials -- every reading where suppression was even possible.

HonestyWhat this does not prove

• Any injection phrasing other than the one shipped. A note claiming the case was already followed up by medical information, claiming it duplicates a case already reported, or written to look like a system banner is a different experiment.
• The reverse direction -- a note ASSERTING that something happened to a patient when the account says nothing of the kind. Over-routing is the cheap direction and was not fired.
• Any injection carried inside the CALLER'S OWN WORDS. That is the section this kit reads most closely and it was not attacked.
• Whether the 2 answers that moved are injection or run-to-run variance. Both moved toward this kit's own answer key on readings the scored run had declined, which is the shape of instability rather than of an attack succeeding -- and nothing here separates the two, because repeat variance was not measured.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never code a complaint of record, make an adverse-event determination, decide reportability, open or close a complaint, release or reject a batch, or notify anybody. This kit produces a PROPOSAL and nothing else.

Stated to the model on every call in src/prompt.py's INSTRUCTION and printed on every record as the Coding authority field; enforced by ABSENCE in the repository, and that absence is SCANNED by evals/check_labels.py before any run may spend.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is a PROMPT RULE PLUS A STATIC SCAN, not a runtime enforcement layer. Nothing stops a forker adding such a path tomorrow; the scan catches it only the next time somebody runs check_labels.py, which is a manual step.
• The injection result is ONE SENTENCE against ONE model on ONE corpus over 17 paired trials. It is not a resistance rate for prompt injection in general, and the 95 pct ceiling of 17.65 pct is printed beside it precisely because a zero is not a proof.
• ⚠︎ ae_content IS NOT AN ADVERSE-EVENT DETERMINATION and this kit never makes one. It is a reading of a text -- does this account describe something that happened to a person -- offered as a ROUTING signal for a human triager. The determination is pharmacovigilance's act on the whole case.
• The proposed code is an OPINION. A QA specialist governs the code of record, and the coordinator-confirms / QA-governs split is not configurable anywhere in this repository.
• The scan reads for acts that reach outside the process. It does not and cannot prove the kit is side-effect free in general.

WatchedWhat is watched, and why that one

The kit measures 103 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

16 measured by the latest run 87 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (ae_present_cells 13, ae_unclear_cells 4, amendment_cells 4, documents 50, escalate_cells 15, logged_code_wrong_cells 25, narrative_only_cells 38, no_escalate_cells 35, no_tree_cells 4, quiet_cells 33, readings 50, readings_scored 50, route_cells 17, scope full, scope_intended 50, stateless False, tie_break_cells 4, words_blind False) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

6 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

triage · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 3 runs measured the same system — they differ on floor, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

triage · with the model in the path — 2 runs. Columns here are only ever compared with each other.

not a time series No two of these 2 runs measured the same system — they differ on stateless, words_blind — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 16 chips that all say so.

DeviationsWhat deviated

0 breaches across 6 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

ON DEMAND, ONCE PER COMPLAINT -- there is no clock and no cursor. A complaint is logged once and read once; nothing here polls, schedules or carries state between readings.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, no framework dependency, and a requirements.txt that names nothing because nothing is needed.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear: the reader -> src/prompt.py -> src/adapters -> evals/scoring.py. No agent, no tool loop, no retrieval step, no state.

The other sideWhat a framework costs you

• You write the retry and timeout policy yourself. That is the point on this kit -- the two values are set from the published ceiling and changed in the same edit as it.
• You write the provider adapters yourself. Two are shipped, in about 40 lines each.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-complaint-code on the fast tier, with the caller's own words, 2026-08-26. This kit records telemetry no telemetry captured — 0 of the 6 readings on this axis, 0 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 4 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

No band column. None of these readings has an alarm band derived for it yet, so there is nothing for the guardrail board to judge them against. Measured and unjudged is a real state and it is stated once; a column of “no band” is not a band.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — one report is one event, whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: ADAPTER_TIMEOUT_S, API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-field exact match against the answer key

Did the arm answer the six cells the way the coding tree settles them? Exact match, trimmed and case-folded on the five enum columns and case-sensitive on the owner, over every gold row -- so a record the arm never answered scores as six misses rather than disappearing from the denominator.

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold. complaint_code, ae_content and the tie-break are stated once per case; category, safety_route and escalate are DERIVED by src/tree.py -- the same functions the free floors and this scorer call -- so the key cannot drift from the rules under test. Every keyed node is asserted available for its product family before anything is keyed.

How tight can the band be? No tolerance band -- a cell matches the key or it does not. And the denominator is 50, so one reading is 2.0 percentage points; the named slices below are 4 readings each and one moves them by 25.

Cadence: Re-score on any change to src/tree.py's NODES, to src/prompt.py, or on any corpus regeneration.

The decisionWhen to reach for it

In one line⚑ The severe column -- an account that never reached safety

Of the readings whose answer key routes to safety, how many did the arm NOT route -- and, on its OWN denominator, of the readings that do not route, how many did it route anyway. The two are never meaned.

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the answer key's safety_route, itself derived by src/tree.route_call() from ae_content -- which routes on PRESENT and on UNCLEAR, deliberately and asymmetrically.

No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? The routed denominator is 17 readings, so one miss is 5.9 percentage points and no band finer than that is meaningful. The PRESENT slice is 13 and UNCLEAR is 4 -- both printed with their denominators every time, and both to be read as directions rather than rates.

Cadence: Re-score on any change to src/tree.route_call(), to the LAY_TERMS list in evals/baseline.py, or on any corpus regeneration.

The decisionWhen to reach for it

In one lineThe escalation, and the tie-break scored as one cell

Did the proposal reach a QA specialist when the tree did not settle it -- and on the readings where two branches fit equally, did the arm get the node AND the escalation right at the same time.

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the answer key's escalate, derived by src/tree.escalate_call() from the proposed node, the adverse-event reading and the case's declared tie-break.

No true/false rates for this grader. It records 3 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? The tie-break denominator is 4 readings. One reading is 25 percentage points and no band finer than that exists. It is printed with its denominator every time it appears.

Cadence: Re-score on any change to TIE_BREAKS or escalate_call() in src/tree.py.

The decisionWhen to reach for it

In one lineDid the arm simply repeat the intake agent's pick -- the diagnosis, not a score

Of the readings whose logged reported-problem code the coding tree does NOT settle on, how many did the arm answer with that same code. It separates 'got it wrong' from 'did not look', and those need different fixes.

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the answer key's logged_code_correct, written by tools/build_corpus.py from the case declaration rather than inferred.

No true/false rates for this grader. It records 1 operating row and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? A count over 25, not a rate to band. Both are printed.

Cadence: Re-score on any change to src/prompt.py or src/select.py -- the two places that decide whether the account reaches the model at all.

The decisionWhen to reach for it