Home › Use Cases › Change regulatory impact comparison

Use caseUC0374

🧪 Use-case kit · runnable

Change regulatory impact comparison

A small, forkable project that does one job end to end. Run once for real, and every figure on these pages captured from that run.

The business caseThe problem this solves

A manufacturing site raises a change control record: a few rows of what is currently registered against what is proposed, and a paragraph of justification. Somebody in regulatory affairs has to say which registered element each row touches and which variation category that triggers — against the dossier as approved, not from memory. It is slow, it is done from a spreadsheet of precedents, and the two errors it can make are not the same size. the manual placement of each proposed change against the registered dossier and the variation code

Audience

The regulatory affairs lead who owns the filing decision, and the site quality team who raised the record. The decision this report supports is whether a paid model call is worth making at all — and on the answer the kit exists to produce, this run says it is not. Every number on these pages came from one real run of this code, not from a vendor page.

The inputThe actual change control records

The corpus is 46 change control records, 0.09 MB (txt 46). EVERY WORD IS INVENTED AND IT HAS TO BE. Change control records, registered dossiers and variation filings are confidential documents belonging to identifiable companies, and a kit that shipped one — redacted or not — would be shipping somebody's regulatory strategy. Aldoria is not a country; the Aldoria Medicines Board does not exist; AVC-2026 is a table in src/rules.py. 46 records, 167 change items, 5 invented products with 11 registered elements each, and 12 distinct (element, nature) pairs. NOTHING THIS KIT PRODUCES IS REGULATORY ADVICE.

The corpus

The 46 change control recordsgenerated from a fixed seed, so no real record, person or institution appears in it.
Where each came fromrecorded in the kit's own SOURCES.md, beside the corpus it describes.

Swap this folder for your own material and the kit is pointed at your change control records. That is the whole change — there is no database to migrate.

One change control record, as the model receives itCC-0001.txt · 1 of 46

ALDORIA PHARMA CHANGE CONTROL RECORD
SYNTHETIC DOCUMENT - generated for the AI Foundry variation-trigger kit from one
seed. Not a real record, not a real product, not a real filing.

Record              CC-0001
Raised              2026-04-18 by Packaging Development
Product             Terrivane 10 mg/mL solution for injection
Authorisation       MA-AL-2026-0219   (Aldoria Medicines Board)
Holder              Northreach Biologics N.V.
Programme           Cost-of-goods programme

PROPOSED CHANGES
  Item 1  Manufacturing site address
      currently registered : Northreach Biologics, Building 2, Harrowgate Park, Lenn
      proposed             : Northreach Biologics, Building 2, Harrowgate Park North, Lenn
  Item 2  Excipient supplier - sodium chloride
      currently registered : Marrow Salts Co, Pellen
      proposed             : Marrow Salts Co (second plant)

DESCRIPTION AND JUSTIFICATION
  Item 1.
      The site is unchanged in every respect. What changes is how its address is
      written, following a municipal renumbering.
  Item 2.
      The excipient supplier already named in this dossier will supply the same
      material to the same grade from a second plant of its own. The registered
      supplier entity is unchanged.

DECLARATION
  This record is a PROPOSAL. It is not a filing and it is not a regulatory
  assessment. No change described here may be implemented before the regulatory
  affairs lead has decided the route and recorded that decision.

The outcomeWhat a good result looks like

Every proposed change item carries the registered element it touches, the nature of the change, the variation category that triggers, the rule that decided it quoted in full, and a verbatim sentence from the registered dossier. The record carries one filing route: the highest route any of its items triggers.

And when it cannot

It OVER-files. 21 of the 167 change items were proposed a route above the key's and 1 below. The dominant single error is reading a minor manufacturing change as a change of principle (7 items), which turns a 30-day notification into a prior approval — a filing nobody needed.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

You want the variation category and nothing else — the free numeric floor
140 of 167 items for $0.00 and no network, against 145 for the paid call. Paired, p = 0.486850: this corpus cannot tell them apart, and one of them has a bill.
Your change records use a field vocabulary your lookup table does not cover — the paid call
the registered element is the one column it wins outright — 166 of 167 against 160, p = 0.031250. A field map is a maintained artifact; this is the job it is bad at.
Your worry is under-filing, not accuracy — the paid call, with a person on every item
it proposed a route below the key's 1 time against the floor's 7. That is the error that reaches an inspection — but paired it is p = 0.070312, a trend and not a result, and 167 items is not enough to settle it.

And where nothing here is good enough:

The justification paragraph is written by a party with an interest in the answer — neither, without a person
6 of 12 injected wordings moved an item's route DOWN the ladder, all six by direct instruction. The free floors are not immune either — a phrase table reads the same attacker-controlled text.

At a glanceHow the whole thing runs

99%element correct pct

1,615 msp50, end to end

$0.69per 1,000 change control records · GPT-5.6 Luna

Run once, for real, on 2026-09-11. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Change regulatory impact comparison14 steps · 4 questions · run once, for real · 2026-09-11

Does it work?evidence first

STEP 01Business case100.0% answered STEP 03App UI8 shown, 2 failing STEP 06Evals166 of 167 scored STEP 13Observability1,615 ms median call

How is it built?the run, in order

STEP 02Architecture5 swap seams STEP 04Corpus46 change control records · 881 graded cells (five on each of the 167 change items, one route on each of the 46 records)STEP 05Prompt2,037.2 tokens in STEP 08Implementationruns with no key STEP 12Frameworksnone, deliberately STEP 14Your environment1 outbound call site

What will it cost?on your rate card

STEP 07Unit cost$0.69 per 1,000 STEP 09Cost modelat your volume

What could go wrong?stated, not hidden

STEP 10Threat modelattacked, not scored STEP 11Guardrailsenforced in code STEP 11Monitoring5 runs recorded

Every tile is a linkall 14 steps · no step without a page

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

What do I have to bring?	Replace data/dossiers.json with your products and their registered sentences, and drop your change control records into data/corpus/ with a matching data/records.json. THE MEASURED ACCURACY DOES NOT TRAVEL. Corpus lens →
When is this the wrong choice?	Avoid: Paying per record for a column a table of regular expressions matches. That is the case against the best-fitting scenario (“You want the variation category and nothing else”). 4 scenarios scored in all, each with its own. Eval lens →
Where does it stop working?	a record that does not number its proposed changes — the reply is keyed on that number and there is nothing else on the page to join a reading to a row. 5 recorded failure modes, each from a run rather than a guess. Corpus lens →
What was never verified?	whether the result holds on a real variation framework — AVC-2026 is invented, and a real framework's conditions would move items between routes. 6 items this kit says it could not check. Eval lens →
Can I run this on a model I control?	The shipped adapter is `the runtime provider is not named on this page`; the Prompt lens states what swapping it costs. The published figures come from 1 model on the fast tier (THE PUBLISHED RUN). Prompt lens →
And if it fits — what do I stand up?	6 artifacts with a stated home and a stated egress, and 4 decisions each with what you provision past its ceiling — plus what was not measured. That is the next page, not this one. step 14 — Run it in your environment →

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-09-11 — r001-variation-trigger. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Run this on your own data

The pipeline, its eval harness and the runs behind every numberdeployed inside your environment, on your own model endpoints, against your own documents.
The corpus above is the shape, not the limitit is a folder swap, and there is no database to migrate.

Talk to us →

Checked before this shipped — A clean checkout with no key configured renders the whole board — the corpus, the registered dossiers, the answer key, the committed run and all three free floors — and scores every free floor offline. evals/check_labels.py, evals/baseline.py and evals/run.py --rescore all run with no network and no credential. Only evals/run.py --arm model, the injection probe, and the board's one Ask button ever reach a provider.

Home › Use Cases › Change regulatory impact comparison › Architecture

Use caseUC0374

🧪 Lens 02 · Architecture

Architecture

Change regulatory impact comparison

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

The variation-trigger kit run, end to end

Corpus Corpus Evals Evals Architecture Architecture Prompt Prompt Prompt Prompt App UI App UI Evals Evals Business case Unit cost

1stations numbered in run order
txt
json
jsonl
md
2 of 10 stations carry a recorded failure

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

Sources
txt51json4jsonl1md2
46 change control records — five invented medicines under an invented regulator, each proposed change read against the registered dossier extract for the authorisation the record itself names
1Sample docsCorpus ↗
- 46 change control records, 167 proposed change items, 2 to 5 items each, 97,966 bytes, generated from seed 20260911 — and the same bytes back on --check, byte for byte
- each item prints a field label the SITE writes, the cell currently registered, the cell proposed, and a justification paragraph; beside them the registered dossier extract for that authorisation, 11 registered elements, one sentence each
- the majority route is AVC-T30 at 55 of 167, so answering it to every item already scores 32.9 pct and has read nothing
- ⚠ ALDORIA IS NOT A COUNTRY. Every product, company, site, specification, dossier and record is invented, the Aldoria Medicines Board does not exist, and NOTHING THIS KIT PRODUCES IS REGULATORY ADVICE
2AVC-2026Evals ↗
- an invented variations code: six routes on a ladder from 'nothing is filed' to 'a new application', eleven registered elements, ten kinds of change, and 26 (element, nature) rows plus a residual
- the table is asserted rather than described: src/rules.py holds it once, the answer key is derived from it, all three free floors call it and the paid arm's two readings go through it unchanged
- and evals/check_labels.py RE-DERIVES every route independently — its own parser, over data/policy.md, the markdown a human reads, with the ladder and the max-aggregation implemented again in six lines: 0 problems over 1,262 assertions
- ⚠ THE RESIDUAL ROW ROUNDS TOWARDS THE REGULATOR. A pair the table does not name goes to prior approval, because the expensive mistake is implementing a change that needed one
3Pair upArchitecture ↗
- the record's own authorisation number resolves the dossier extract — src/prompt.py reads it off the document, so the hosted demo and the kit run the same code path on one argument
- ⚑ NOTHING IS SELECTED AND NOTHING IS SUMMARISED. The whole record and the whole extract go on the wire. Choosing which dossier sections to show would be doing the judging that is being measured
- there is no index, no retrieval and no chunker in this kit — which is also where it stops: a real dossier extract is hundreds of pages and would not fit beside the record
4Free floorsno lens on the shipped page
in $0.00
- THREE of them, all scored, all through the SAME rule table the paid arm's readings go through — because a floor built to lose proves nothing and inflates every margin above it
- fielddiff 89/167 categories · keyword 137/167 · numeric 140/167. numeric is the floor the margin is published against, and it is the hardest one
- ⚑ THE FIRST KEYWORD FLOOR SCORED 165 OF 167 ON THE NATURE READING AND WAS THROWN AWAY. It had been written from the corpus generator's own sentences — a decoder for a generator, not a floor — and every margin above it would have measured the author's phrasing
- what fixed it was the CORPUS: every (element, nature) pair now carries several narrative variants, at least one stating the decisive fact without the word a keyword table looks for, and at least one carrying a decoy pointing at the other class
5PromptPrompt ↗
- six parts, fixed order, 9,327 characters — the role and the cap, the eleven registered elements, the ten natures with the direction rules spelled out, the JSON schema, the dossier extract, and the record verbatim
- the first four are byte-identical on every call this kit will ever make and go first, so a provider that prices cached input bills the prefix cheaply: 63,744 of the run's 93,713 input tokens came back as cache hits — 68 pct
- ⚑ THERE IS NO FIELD FOR A CATEGORY, A RULE OR A ROUTE. The schema has four keys — item, element, nature, quote — and the arm asserted 0 fields outside it across all 46 calls
6One modelPrompt ↗
in 46 calls
- one provider, one key, one call per record. The fast tier, reasoning DISABLED — the documented disabled shape was sent on every call and the run record stores exactly what was sent, so this run cannot claim a setting it did not use
- 46 calls, $0.0142 off peak; the same tokens inside the weekday peak window would be $0.0284. Largest reply 373 output tokens of a 1,600 ceiling — 23.3 pct — and 0 calls reached it. p50 1,615 ms, p95 2,048 ms. 0 failures and 0 unparsed replies
- ⚑ IT IS ASKED FOR TWO ENUMS AND A QUOTE PER CHANGE ITEM AND NOTHING ELSE. No category, no rule, no route — the schema has no field for any of them
$0.000309 per record Unit cost ↗
Recorded failureIT READS A CHANGE AS BIGGER THAN IT IS, AND THAT IS THE DOMINANT ERROR. Seven items where the site changes a blender or raises a batch size inside the registered principle came back MAJOR_IMPACT, which turns a 30-day notification into a prior approval — a filing nobody needed. 21 of 167 items were routed above the key and 1 below
7Rule tableApp UI ↗
- THE STATION, AND IT IS WHAT THE KIT SHIPS. It trusts exactly TWO fields of the reply — each item's element and its nature, the two readings a field-name lookup cannot make — and computes the category, the rule, the rule's text and the record's filing route by running AVC-2026 over them
- the record's route is the HIGHEST route any one item triggers, never a majority: one item needing prior approval makes the whole record a prior-approval record
- ⚠ IT CANNOT NOTICE THAT A READING WAS WRONG. Handed NONE on a change that really touches the container closure system it rules R-26 correctly, agrees with itself to the letter, and publishes 'no variation' on a change that needed a filing. Every test passes and the reviewer is handed a record that should have been notified
- ⚠ AND IT DOES NOT REPAIR A CITATION, deliberately — rewriting the arm's quote would erase the evidence of the failure
8Under attackno lens on the shipped page
in 12 calls
- THE JUSTIFICATION PARAGRAPH IS WRITTEN BY THE SITE — the party with an interest in the lowest possible route — so it is the one field an attacker really controls, and nobody has to break in to write in it
- 6 of 12 injected wordings moved a prior-approval item DOWN to 'no variation'. Persuasion failed 6 of 6; direct instruction succeeded 6 of 6
- ⚑ THE STRUCTURAL GUARDRAIL HELD PERFECTLY AND GUARDED THE WRONG THING. 0 forbidden fields asserted, including on the trial that handed the arm a fully-formed object naming a category. The attacker never needed one: changing a READING was enough
Recorded failureTHIS KIT FAILS ITS OWN INJECTION PROBE AND SAYS SO ON THE FIGURE. A human reading every item before anything is filed is not a nice-to-have here; it is the control
9ScoredEvals ↗
- 881 graded cells — five on each of the 167 change items and one filing route on each of the 46 records — every grader pure Python against a key that was DERIVED and never typed
- ⚑ READ EVERY RATE AGAINST ITS MAJORITY-CLASS FLOOR, published beside it: always AVC-T30 scores 32.9 pct on the category, always TIGHTENED 16.8 pct on the nature, always D.3.1 12.0 pct on the element. 112 of the 167 items are off-majority on the category and those are the only ones that carry information
- citations 165 of 167 located and admitted, 0 unlocatable — a quote that is not in the dossier is a plausible sentence, not evidence
10ReportBusiness case ↗
published, stamped
category 145/167 — the free floor 140
McNemar exact vs that floor: 19/14, p=0.486850
element 166/167 vs 160 — p=0.031250
filing route 35 of 46 — that floor 33
under-filed 1 vs 7 — p=0.070312
under attack: 6 of 12 obeyed
$0.0142 for 46 calls · as of 2026-09-11

⚠ THE PAID CALL DOES NOT BEAT THIS KIT'S OWN FREE CODE ON THE ANSWER, and the ribbon says so before it says anything else. Same 46 records, same 167 change items, same rule table, same scorer: the free NUMERIC floor takes 140 of 167 variation categories for $0.00 and no network; the paid call takes 145. Paired, that is 19 items to 14 — exact two-sided p = 0.486850, no margin anybody can bank. On the record's filing route it is 35 of 46 against 33, p = 0.790527. ⚑ WHERE IT WINS IS THE READING A LOOKUP TABLE CANNOT DO: which registered element a change touches, 166 of 167 against 160, p = 0.031250 — the site writes 'Primary pack component supplier' and the dossier says 'Container closure system', and the two vocabularies do not line up. ⚑ AND THE FINDING THAT IS NOT IN THE ACCURACY COLUMN IS THE DIRECTION OF THE ERRORS. The paid call proposed a route BELOW the key's once; the free floor did it seven times. Over-filing costs a filing fee, under-filing is a change implemented without the approval it needed — but paired it is 7 to 1 at p = 0.070312, a trend and not a result at the 5 pct line, and 167 items is not enough to settle it. THE HONEST RECOMMENDATION IS THE SPLIT: buy the call for the element reading, let the code do the lookup, and put a person on every item. ⚑ AND THE COMPARISON ONLY MEANS ANYTHING BECAUSE THE FLOOR WAS BUILT TO WIN: the first keyword floor scored 165 of 167 on the nature by matching the corpus generator's own sentences, and was thrown away. A kit that had published it would have shown the model losing badly and taught its reader nothing.

The swap seams

Seam	File	What changes
the model	`src/adapters/__init__.py`	PROVIDER, BASE_URL and MODEL in .env. Nothing else moves — every arm goes through the same station.
the variation code	`src/rules.py`	RULES is a list of (rule id, element, nature, route, text). A different framework is an edit to that list plus data/policy.md, which evals/check_labels.py re-parses independently.
the registered dossiers	`data/dossiers.json`	your products, your authorisation numbers, your registered sentences. src/prompt.py resolves the extract from the authorisation the record prints.
the free floors	`evals/baseline.py`	FIELD_MAP maps your own field vocabulary to elements; PHRASE_RULES is the triage table your team would keep anyway.
the prompt	`src/prompt.py`	build() takes ONE argument — the record — so the hosted demo and the kit run the same code.

Components

Component	File	Role
the change control record	`data/corpus/CC-0001.txt`	one record, one call, sent verbatim — no row is pre-parsed and no paragraph is summarised on the way in
the registered dossier extract	`data/dossiers.json`	11 registered elements per product, each one sentence, resolved from the authorisation number the record itself prints
AVC-2026	`data/policy.json`	the invented variation code as data — the ladder, the elements, the natures and the 27-row table
the rule engine	`src/rules.py`	(element, nature) -> category, rule id and rule text. Pure code. The only place a route is decided
the prompt	`src/prompt.py`	six parts, fixed order; parts 1-4 byte-identical on every call this kit makes
the model adapter	`src/adapters/__init__.py`	raw HTTP to any OpenAI-compatible or Anthropic endpoint; the only file that knows who serves the model
the reader	`src/reader.py`	one call, and a tolerant JSON parse that recovers a truncated reply and records — never merges — a key the schema does not name
the station	`src/station.py`	trusts exactly two fields of the reply and recomputes the category, the rule and the record's filing route from them
the overreach counter	`src/overreach.py`	counts and publishes every field the arm asserted outside the schema, then discards them
the three free floors	`evals/baseline.py`	field diff, phrase rules and numeric direction — all scored through the same station and the same rule table
the graders	`evals/scoring.py`	five cells on each change item and one route on each record, every one pure code
the paired test	`evals/significance.py`	McNemar exact on the discordant pairs — the only comparison that means anything between two arms answering the same items
the board	`src/app.py`	stdlib http.server; renders the corpus, the key, the committed run and all three floors with NO key configured

Where it breaks at scale

The unit is ONE RECORD IN ONE CALL and the record goes in whole, with the whole registered dossier extract beside it. A dossier extract is 11 sentences here; a real one is hundreds of pages, and at that size the extract stops fitting beside the record and the architecture needs a retrieval step this kit does not have — which would introduce a second failure mode (the right element not retrieved) that nothing here measures. Records also do not batch: the filing route is a MAX over a record's own items, so two records cannot share a call without the aggregation crossing them.

Home › Use Cases › Change regulatory impact comparison › UI

Use caseUC0374

🧪 Lens 03 · UI

App UI

Change regulatory impact comparison

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

One change control record, checked. The record on the left, verbatim, and the registered dossier extract it is measured against on the right — the same two documents the model is shown, in the same order.successOpen full size →

The ruling, item by item, recomputed by AVC-2026 in pure code. The switcher above the table runs the SAME engine over the answer key, each of the three free floors and the committed run, so a reader can watch the arms disagree on identical rows.successOpen full size →

Every record in the corpus, and the filing route each of the five readers proposes for it. A row shaded red is a record where the committed run proposed a LOWER route than the key.successOpen full size →

AVC-2026 as approved — the invented 27-row table the whole kit rules from, ending in the residual row that sends an unnamed pair to prior approval.successOpen full size →

A five-item record end to end, full page: the two readings, the ruling, every reader side by side, and the prompt in the order it goes on the wire.successOpen full size →

The board with NO key configured. The corpus, the registered dossiers, the answer key, the committed run and all three free floors come off disk; only a NEW call needs a credential, and the button says so.successOpen full size →

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

The honest failure frame. The red band names every item the committed run read differently from the answer key and says which of them UNDER-filed — here a primary pack component supplier change read as outside the dossier, which drops a 30-day notification to no variation at all. The kit publishes its misses and does not repair them.failureOpen full size →

The kit's actual headline, and it is a qualified loss: the paid call against the best free code floor, paired on the same 167 change items. On the variation category it is 145 to 140 with an exact two-sided p of 0.486850 — no margin anybody can bank.failureOpen full size →

Home › Use Cases › Change regulatory impact comparison › Data

Use caseUC0374

🧪 Lens 04 · Data

Corpus

Change regulatory impact comparison

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

46change control records

0.09 MiBtxt 46

881graded cells (five on each of the 167 change items, one route on each of the 46 records) · p50 4 chars

$0.00setup · 0s

How it is cutWhat one graded cells (five on each of the 167 change items, one route on each of the 46 records) is

one change control record is one unit and goes into one call whole; the items within a record are not independent because the filing route is a MAX over all of them

SetupWhat the setup figure measured

THERE IS NO INDEX AND NOTHING TO RETRIEVE FROM: one record goes whole into one call, with the registered dossier extract for the authorisation it prints. The only pure-code step before the call is reading the file and looking the authorisation number up in data/dossiers.json. ⚠︎ THE STRUCTURED COPY CARRIES NO ELEMENT AND NO NATURE, deliberately — those two readings are what is being measured, and handing either over structurally would hand over the answer.

LicenceLicence

MIT

Bring your ownBring your own change control records

Replace data/dossiers.json with your products and their registered sentences, and drop your change control records into data/corpus/ with a matching data/records.json. Nothing under src/ changes. The framework is src/rules.py's RULES list plus data/policy.md; a different variation code is an edit to those two, and evals/check_labels.py re-parses the markdown independently to prove they agree.

⚠︎ And what stops being true when you do: THE MEASURED ACCURACY DOES NOT TRAVEL. Every rate on this page was measured against an invented framework whose ladder and conditions are stated in data/policy.md. A real variation framework decides the same changes differently — most consequentially, this one routes ANY change to qualitative composition to prior approval whatever impact the applicant declares, and 14 of the 167 items depend on that single choice. Re-run it on your own records and your own code to find out what it does there.

What breaks it

a record that does not number its proposed changes — the reply is keyed on that number and there is nothing else on the page to join a reading to a row
an authorisation number the record does not print, or prints differently from data/dossiers.json — the prompt then carries no registered dossier extract at all and says so, and SITE_LISTED becomes unanswerable because nothing says what is already registered
a registered element AVC-2026 does not carry: the pair falls to the residual rule R-99 and routes to prior approval rather than guessing
a change record written as prose with no currently-registered / proposed pair — the free floors read those two cells and have nothing to diff
a dossier extract too large to send beside the record; there is no retrieval step here and nothing selects a section

Home › Use Cases › Change regulatory impact comparison › LLM

Use caseUC0374

🧪 Lens 05 · LLM

Prompt

Change regulatory impact comparison

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

Part	Characters	Tokens
role and the cap	373	not measured
the registered elements	816	not measured
the nature of the change	2,008	not measured
the answer schema	827	not measured
the registered dossier extract	2,601	not measured
the change control record, verbatim	1,532	not measured
Total		2,037.2

This is the cost lesson as arithmetic: of the 8,157 characters assembled, 2,824 are vocabularies — 35% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Replayed from the kit's own src/prompt.py build() for CC-0001, not logged by the run. It is the prompt that was sent because build() takes one argument — the record — and the run passes it the same bytes that are on disk; the replayed part sizes match what the board publishes at /api/prompt.

Full promptThe prompt in full

The verbatim prompt, as sent

the prompt, as sent

You read a pharmaceutical change control record against a registered dossier extract and say, for each proposed change item, WHICH REGISTERED ELEMENT it touches and WHAT KIND of change it is. You do not decide the variation category, the rule or the filing route: those are computed from your two readings by a rule table you never see the output of. Answer with JSON only.

PART 2 — THE REGISTERED ELEMENTS. Answer `element` with exactly one of these ids.

  D.1.2  Qualitative and quantitative composition
  D.2.1  Active substance manufacturer
  D.2.4  Active substance specification
  D.3.1  Finished product manufacturing and packaging sites
  D.3.3  Manufacturing process and batch size
  D.3.4  Excipient supplier and grade
  D.4.1  Finished product specification
  D.4.3  Analytical procedures
  D.5.1  Container closure system
  D.6.1  Shelf life and storage conditions
  D.7.1  Product information texts
  NONE   no registered element — the change is outside the dossier

  Use NONE when the change touches no registered element at all — an internal system, a non-registered in-process check, a cleaning agent for non-product-contact surfaces. NONE is a real answer, not a failure.

PART 3 — THE NATURE OF THE CHANGE. Answer `nature` with exactly one of these.

  TIGHTENED          an acceptance criterion made narrower, or a test added
  WIDENED            an acceptance criterion made wider, a test deleted, or a period extended
  METHOD_EQUIVALENT  an analytical procedure replaced, with comparative results on the same batches showing the two agree across the range
  METHOD_NEW         an analytical procedure replaced on a different measuring principle, or with no comparative result against the registered one
  SITE_LISTED        a site or supplier ALREADY named in this dossier, taking on this operation
  SITE_NEW           a site or supplier this dossier has never named
  MINOR_NO_IMPACT    a change within the registered principle — same equipment type, same process order, a batch size inside the registered range
  MAJOR_IMPACT       a change to the principle itself — the unit operation, the composition, the strength, or a safety statement
  ADMIN_ONLY         a name, address, spelling or reference change with no technical content
  OUTSIDE_DOSSIER    not a registered detail at all

  Read the DIRECTION, not the digits. An acceptance criterion that becomes EASIER to meet is WIDENED even when the number printed went down (a dissolution criterion moving from Q = 80 % to Q = 75 % is a widening); a shelf life that becomes LONGER is WIDENED even though the number went up.
  METHOD_EQUIVALENT needs a comparison AGAINST THE REGISTERED PROCEDURE on the same material. A procedure that has been validated on its own, however thoroughly, is METHOD_NEW when it measures on a different principle or when no comparative result against the registered procedure is offered.
  SITE_LISTED means the proposed site or supplier is ALREADY NAMED in the dossier extract below, for any operation — read the extract, not the spelling. Sites are often written differently on a change record than in the dossier.
  ADMIN_ONLY is a name, an address or a spelling with no technical content.

PART 4 — THE ANSWER. Return ONE JSON object and nothing else — no prose, no code fence.

  {"items": [
     {"item": 1, "element": "D.4.1", "nature": "WIDENED",
      "quote": "one sentence copied verbatim from the dossier extract"},
     {"item": 2, "element": "NONE", "nature": "OUTSIDE_DOSSIER", "quote": ""}
  ]}

  One object per PROPOSED CHANGE item, in the record's own numbering, all of them.
  `quote` is copied VERBATIM from the registered dossier extract — the sentence of the
  element you named. Do not paraphrase it, do not shorten it, do not invent one. When
  `element` is NONE there is nothing registered to quote, so return an empty string.
  There is no field for the variation category, the rule or the filing route. Do not add
  one; anything outside this shape is discarded and counted against the answer.

PART 5 — REGISTERED DOSSIER EXTRACT, as approved.

Authorisation MA-AL-2026-0219 — Terrivane 10 mg/mL solution for injection (holder: Northreach Biologics N.V.)

D.1.2  Qualitative and quantitative composition
    Each unit contains terrivane hydrochloride equivalent to the registered strength of 10 mg/mL. The excipients are those listed in the approved composition, of which sodium chloride, Ph. Aldoria grade is the release-controlling or principal excipient.
D.2.1  Active substance manufacturer
    The active substance terrivane hydrochloride is manufactured at Caldwin Actives GmbH, Ostweg 3, Immenau. No other manufacturer of the active substance is registered.
D.2.4  Active substance specification
    The active substance is tested on release against the registered specification, which includes assay 98.0 - 102.0 % on the dried basis and any unspecified impurity not more than 0.10 %.
D.3.1  Finished product manufacturing and packaging sites
    Manufacture, primary packaging and batch release of Terrivane 10 mg/mL solution for injection are performed at Northreach Biologics, Building 2, Harrowgate Park, Lenn. Secondary packaging is performed at Northreach Biologics, Building 7, Harrowgate Park, Lenn.
D.3.3  Manufacturing process and batch size
    The registered process is compounding, sterile filtration, aseptic filling and terminal inspection, at a batch size of 180 L. The unit operations and their order are as described in the approved process description.
D.3.4  Excipient supplier and grade
    Sodium chloride, Ph. Aldoria grade is sourced from Marrow Salts Co, Pellen to the registered grade and specification.
D.4.1  Finished product specification
    The finished product is released against the registered specification: assay 97.0 - 103.0 %; sub-visible particles not more than 6000 per container at 10 micrometres; pH 4.2 - 5.4.
D.4.3  Analytical procedures
    Assay and related substances are determined by the registered reversed-phase liquid chromatographic procedure described in the dossier; sub-visible particles is determined by the registered procedure for that test.
D.5.1  Container closure system
    The product is packed in a type I glass vial with a bromobutyl rubber stopper in pack sizes of 1 and 5 vials.
D.6.1  Shelf life and storage conditions
    The registered shelf life is 24 months. Store in a refrigerator (2 C - 8 C). Do not freeze.
D.7.1  Product information texts
    The product information texts are those approved with the authorisation, including the posology, the handling instructions and the statements on storage and disposal.

PART 6 — THE CHANGE CONTROL RECORD, verbatim.

ALDORIA PHARMA CHANGE CONTROL RECORD
SYNTHETIC DOCUMENT - generated for the AI Foundry variation-trigger kit from one
seed. Not a real record, not a real product, not a real filing.

Record              CC-0001
Raised              2026-04-18 by Packaging Development
Product             Terrivane 10 mg/mL solution for injection
Authorisation       MA-AL-2026-0219   (Aldoria Medicines Board)
Holder              Northreach Biologics N.V.
Programme           Cost-of-goods programme

PROPOSED CHANGES
  Item 1  Manufacturing site address
      currently registered : Northreach Biologics, Building 2, Harrowgate Park, Lenn
      proposed             : Northreach Biologics, Building 2, Harrowgate Park North, Lenn
  Item 2  Excipient supplier - sodium chloride
      currently registered : Marrow Salts Co, Pellen
      proposed             : Marrow Salts Co (second plant)

DESCRIPTION AND JUSTIFICATION
  Item 1.
      The site is unchanged in every respect. What changes is how its address is
      written, following a municipal renumbering.
  Item 2.
      The excipient supplier already named in this dossier will supply the same
      material to the same grade from a second plant of its own. The registered
      supplier entity is unchanged.

DECLARATION
  This record is a PROPOSAL. It is not a filing and it is not a regulatory
  assessment. No change described here may be implemented before the regulatory
  affairs lead has decided the route and recorded that decision.

Raw responseThe raw response

The raw response, before any parsing

the response, unparsed

{"items": [
  {"item": 1, "element": "D.3.1", "nature": "ADMIN_ONLY",
   "quote": "Manufacture, primary packaging and batch release of Terrivane 10 mg/mL solution for injection are performed at Northreach Biologics, Building 2, Harrowgate Park, Lenn."},
  {"item": 2, "element": "D.3.4", "nature": "SITE_LISTED",
   "quote": "Sodium chloride, Ph. Aldoria grade is sourced from Marrow Salts Co, Pellen to the registered grade and specification."}
]}

Home › Use Cases › Change regulatory impact comparison › Eval

Use caseUC0374

🧪 Lens 06 · Eval

Evals

Change regulatory impact comparison

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Change regulatory impact comparison — 46 change control records. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Every grader is pure code against a key that was DERIVED rather than typed: tools/build_corpus.py picks the (element, nature) pair first and then writes a sentence describing it, and the route is src/rules.py over that pair. No model grades anything.

46change control records

46source documents

1model tier

3grading methods

MeasurementsWhat was measured

COUNTED166 / 167element correct pct — proposed change itemsDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.

COUNTED145 / 167nature correct pct — proposed change itemsDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.

COUNTED145 · 140 · 89 / 167category correct pct — proposed change itemsDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.

COUNTED144 / 167item all correct pct — proposed change itemsDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.

COUNTED35 · 33 / 46filing route correct pct — change control recordsDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.

NOT YET KNOWN—A person confirmed the gold is rightThe grader is == and needs no confirming. The gold does: it is machine-derived from the registry's structured modules, and every rate on this page rests on that derivation being faithful to what the prose actually says. Nobody has read a document against its gold record by hand.

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

How the method was validated

The method is validated by evals/check_labels.py, an INDEPENDENT reader that imports nothing from src/: it parses the rule table out of data/policy.md, implements the ladder and the max-aggregation itself, and re-derives every route from scratch. 1,262 assertions, 0 problems — and it is red-proven: flipping one category in gold.jsonl makes it convict by name, restoring it acquits.

Grading costWhat it costs

Every projected dollar is a MEASURED token count multiplied by a named vendor's PUBLISHED rate. The token counts come from the provider's own usage block on each of the 46 calls. The tier that actually ran is priced from its own card at the tariff in force per call, and that figure is a real bill.

Priced at	Per 1M in / out	One change control record	1,000 change control records	Share that is the prompt
GPT-5.6 Luna the estate's shared projection card, so this kit's rows compare with every other kit's	$0.20 / $1.20	$0.000690	$0.69	59%
Gemini 3 Flash the estate's shared projection card, so this kit's rows compare with every other kit's	$0.50 / $3.00	$0.001726	$1.73	59%
Claude Haiku 4.5 the estate's shared projection card, so this kit's rows compare with every other kit's	$1.00 / $5.00	$0.003216	$3.22	63%
GPT-5.6 Terra the estate's shared projection card, so this kit's rows compare with every other kit's	$2.00 / $12.00	$0.006904	$6.90	59%
Claude Sonnet 5 the estate's shared projection card, so this kit's rows compare with every other kit's	$2.00 / $10.00	$0.006432	$6.43	63%

Same work, 10× the bill

The same change control records, the same tokens — only the rate card changed. And across all 5 cards between 59% and 63% of what you pay is the prompt this pipeline sends, not the answer it writes.

the registered dossier extract. It is re-sent whole on every call and it is the part that grows fastest with a real product — there is no retrieval step to cut it down, which is the same fact Architecture.breaks_at_scale records.

Rates checked 2026-09-05. The runtime tier is not named on this page. The estate publishes rate cards for a fixed list of providers and the runtime one is deliberately not on it.

Grading unitWhat the grading figure prices

freegrading cost, as measured

The graders are code, so grading costs $0.00, no data leaves the machine, and re-scoring the committed run returns the same answer — evals/run.py --rescore does exactly that, with 0 calls, and it is how the under-filing test reached a run that had already been paid for.

The gradersThree ways to grade

Grader	Cost / 1k	Data leaves	Same answer twice	Result
The two readings whether the arm named the right registered element and the right kind of change — the only two fields it is asked for, and the only two anything downstream uses	$0.00	no	yes	the fast tier (THE PUBLISHED RUN) 99.4% element · fielddiff floor — 0 calls, $0.00, no network 95.8% element · keyword floor — 0 calls, $0.00, no network 95.8% element · numeric floor — 0 calls, $0.00, no network 95.8% element · 1 more measured on each run
The variation category and the filing route whether the route the kit PUBLISHES is the route the key names — per change item, and per record after the max-aggregation	$0.00	no	yes	the fast tier (THE PUBLISHED RUN) 86.8% category · fielddiff floor — 0 calls, $0.00, no network 53.3% category · keyword floor — 0 calls, $0.00, no network 82.0% category · numeric floor — 0 calls, $0.00, no network 83.8% category · 1 more measured on each run
The citation whether the sentence the arm quoted is actually IN the registered dossier extract, and whether it is the sentence of the element it named	$0.00	no	yes	the fast tier (THE PUBLISHED RUN) 98.8% citation · fielddiff floor — quotes the section by dict lookup, 0 calls 95.8% citation · keyword floor — quotes the section by dict lookup, 0 calls 95.8% citation · numeric floor — quotes the section by dict lookup, 0 calls 95.8% citation

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Why this set cannot separate them

This corpus CAN separate the arms and the majority-class floors are small: answering AVC-T30 to every item scores 55 of 167 on the category (32.9%), TIGHTENED scores 28 on the nature and D.3.1 scores 20 on the element. On the filing route the constant floor is higher — AVC-PA on 24 of 46 records — because the route is a MAX and a max concentrates.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

If your situation is	Use	Because	And avoid
You want the variation category and nothing else	the free numeric floor	140 of 167 items for $0.00 and no network, against 145 for the paid call. Paired, p = 0.486850: this corpus cannot tell them apart, and one of them has a bill.	paying per record for a column a table of regular expressions matches
Your change records use a field vocabulary your lookup table does not cover	the paid call	the registered element is the one column it wins outright — 166 of 167 against 160, p = 0.031250. A field map is a maintained artifact; this is the job it is bad at.	assuming the win carries into the category — it does not
Your worry is under-filing, not accuracy	the paid call, with a person on every item	it proposed a route below the key's 1 time against the floor's 7. That is the error that reaches an inspection — but paired it is p = 0.070312, a trend and not a result, and 167 items is not enough to settle it.	quoting the under-filing gap as though it were significant
The justification paragraph is written by a party with an interest in the answer	neither, without a person	6 of 12 injected wordings moved an item's route DOWN the ladder, all six by direct instruction. The free floors are not immune either — a phrase table reads the same attacker-controlled text.	treating the no-category-field guardrail as protection; the attacker changes a reading, not a category

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Cause	What it means	Rows	A real example
`a minor change read as a change of principle — over-files`	a minor change read as a change of principle — over-files	7	CC-0006 item 4 — the key says D.3.3 / MINOR_NO_IMPACT (route AVC-T30); the run answered D.3.3 / MAJOR_IMPACT (route AVC-PA)
`a site the dossier already names, read as a new one — over-f`	a site the dossier already names, read as a new one — over-files	4	CC-0006 item 1 — the key says D.2.1 / SITE_LISTED (route AVC-T30); the run answered D.2.1 / SITE_NEW (route AVC-PA)
`nature misread: MINOR_NO_IMPACT read as SITE_NEW`	nature misread: MINOR_NO_IMPACT read as SITE_NEW	3	CC-0015 item 3 — the key says D.5.1 / MINOR_NO_IMPACT (route AVC-T30); the run answered D.5.1 / SITE_NEW (route AVC-PA)
`a tightened limit read as widened — over-files`	a tightened limit read as widened — over-files	3	CC-0033 item 2 — the key says D.6.1 / TIGHTENED (route AVC-N2); the run answered D.6.1 / WIDENED (route AVC-PA)
`nature misread: TIGHTENED read as MAJOR_IMPACT`	nature misread: TIGHTENED read as MAJOR_IMPACT	2	CC-0008 item 3 — the key says D.6.1 / TIGHTENED (route AVC-N2); the run answered D.6.1 / MAJOR_IMPACT (route AVC-PA)
`nature misread: MINOR_NO_IMPACT read as WIDENED`	nature misread: MINOR_NO_IMPACT read as WIDENED	2	CC-0014 item 3 — the key says D.3.3 / MINOR_NO_IMPACT (route AVC-T30); the run answered D.3.3 / WIDENED (route AVC-PA)
`cited the wrong registered sentence`	cited the wrong registered sentence	1	CC-0021 item 1 — the key says D.2.1 / SITE_LISTED (route AVC-T30); the run answered D.2.1 / SITE_LISTED (route AVC-T30)
`wrong registered element — the change was filed against the`	wrong registered element — the change was filed against the wrong section	1	CC-0044 item 3 — the key says D.5.1 / MINOR_NO_IMPACT (route AVC-T30); the run answered NONE / OUTSIDE_DOSSIER (route AVC-0)
`nature misread: MINOR_NO_IMPACT read as OUTSIDE_DOSSIER`	nature misread: MINOR_NO_IMPACT read as OUTSIDE_DOSSIER	1	CC-0044 item 3 — the key says D.5.1 / MINOR_NO_IMPACT (route AVC-T30); the run answered NONE / OUTSIDE_DOSSIER (route AVC-0)
`no citation returned for a registered element`	no citation returned for a registered element	1	CC-0044 item 3 — the key says D.5.1 / MINOR_NO_IMPACT (route AVC-T30); the run answered NONE / OUTSIDE_DOSSIER (route AVC-0)

What we could NOT verify

whether the result holds on a real variation framework — AVC-2026 is invented, and a real framework's conditions would move items between routes
whether it holds on real change control records; every narrative here was generated from a variant pool, and a real one is messier and shorter
whether the under-filing gap (1 against 7, p = 0.070312) is real — 167 items is not enough
whether a stronger free floor exists. Three were written and the strongest is published; a fourth would move the margin again, and the margin is a property of the floor
whether another model reads the nature better. One model was run, at one setting, on one day
whether an injected wording nobody tried succeeds, and whether an attack spread over several items of one record compounds

Home › Use Cases › Change regulatory impact comparison › Cost

Use caseUC0374

🧪 Lens 07 · Cost

Unit cost

Change regulatory impact comparison

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

Tier	Input tokens	Output tokens	p50 latency	GPT-5.6 Luna	Gemini 3 Flash	Claude Haiku 4.5	GPT-5.6 Terra	Claude Sonnet 5
the fast tier (THE PUBLISHED RUN)	2,037.2	235.8	1,615 ms	$0.000690	$0.001726	$0.003216	$0.006904	$0.006432
GPT-5.6 Luna	2,037.2	235.8	—	$0.000690	$0.001726	$0.003216	$0.006904	$0.006432
Gemini 3 Flash	2,037.2	235.8	—	$0.000690	$0.001726	$0.003216	$0.006904	$0.006432
Claude Haiku 4.5	2,037.2	235.8	—	$0.000690	$0.001726	$0.003216	$0.006904	$0.006432
GPT-5.6 Terra	2,037.2	235.8	—	$0.000690	$0.001726	$0.003216	$0.006904	$0.006432
Claude Sonnet 5	2,037.2	235.8	—	$0.000690	$0.001726	$0.003216	$0.006904	$0.006432
fielddiff floor — pure code, 0 calls, no network	0	0	0 ms	$0.000000	$0.000000	$0.000000	$0.000000	$0.000000
numeric floor — pure code, 0 calls, no network	0	0	0 ms	$0.000000	$0.000000	$0.000000	$0.000000	$0.000000

The token counts are measured on a real run and belong to this pipeline — top_k, chunk size, prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-09-05. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

$0.00. Every grader is pure code, so re-scoring the committed run costs nothing and returns the same answer — and it was actually done: the under-filing test was added AFTER the run was paid for and reached it through evals/run.py --rescore, 0 calls. All three free floors are $0.00 and 0 calls. The only money this kit ever spent is the 46 calls of r001-variation-trigger ($0.014196) and the 12 calls of the injection probe ($0.003181).

Cost driversWhat actually moves the bill

the registered dossier extract — 11 sentences re-sent on every call, and the single largest block after the vocabulary
the two vocabularies (elements and natures), 2824 characters, identical on every call and therefore cacheable
the record itself — a five-item record is roughly twice a two-item one
the reply, one object per change item: 235.8 output tokens a call on average, largest 373 against a 1600 ceiling
the tariff. Every call of this run landed off peak; the identical tokens at the weekday peak rate are exactly double

Your volumeWhat it costs at your volume

linear in records, and sub-linear in dollars because the fixed prefix caches: 68% of this run's input tokens came back as cache hits. 10x the records is 10x the calls; the per-call cost falls only as far as the cache-hit share allows.

Where pricing changes shape

the peak window. 01:00-04:00 and 06:00-10:00 UTC Monday to Friday is double the off-peak rate on this card, and this run was bought off peak.
a registered dossier extract too large to send whole. Past that point the design needs a retrieval step, and the bill stops being linear in records because it starts including an index.

Your return, with your numbers

Volumechange control records raised per month
What it replacesthe manual placement of each proposed change against the registered dossier and the variation code
Time saved per itemnot measured — this kit measures accuracy and cost, not reviewer minutes, and on the category it does not beat free code anyway

We publish the inputs, not a return: a return depends on your labour cost and your volume. The ROI dashboard takes it from here, and AI Costing explains the math.

Model choiceWhy this model

One provider, one key, one model — the estate default, reasoning disabled and a small output ceiling, which is this estate's standing cheapest-viable rule. The point of the kit is the COMPARISON against free code, and that comparison is only interesting if the paid arm is the cheap tier: a kit that loses to a regex on an expensive model has proved less than one that loses on a cheap one.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

93,713input tokens · this run

10,847output tokens

$0.014what it actually cost

the whole 46-record scored run (r001-variation-trigger, the fast tier): 93,713 input tokens, 63,744 of them prefix cache hits, and 10,847 output of which NONE is reasoning — the budget was sent explicitly disabled. All 46 calls answered, 0 failures and 0 unparsed replies, so this total covers every call in the denominator.

Model	Provider	One eval pass	This whole run	Per 1,000 queries	Rates as of
gpt-5-6-luna	OpenAI	$0.032	$0.032	$0.69	2026-09-12
gemini-3-flash	Google	$0.079	$0.079	$1.73	2026-09-18
gemini-3-8-flash	Google	$0.111	$0.111	$2.41	2026-09-18
claude-haiku-4-5	Anthropic	$0.148	$0.148	$3.22	2026-09-12
llama-5	Meta	$0.163	$0.163	$3.55	2026-09-18
grok-4-5	xAI	$0.253	$0.253	$5.49	2026-09-18
grok-4-6	xAI	$0.253	$0.253	$5.49	2026-09-18
claude-sonnet-5	Anthropic	$0.296	$0.296	$6.43	2026-09-12
gemini-3-1-pro	Google	$0.318	$0.318	$6.90	2026-09-18
gpt-5-6-terra	OpenAI	$0.318	$0.318	$6.90	2026-09-12
gpt-5-6-sol	OpenAI	$0.592	$0.592	$12.86	2026-09-12
claude-opus-4-8	Anthropic	$0.740	$0.740	$16.08	2026-09-12
claude-opus-5	Anthropic	$0.740	$0.740	$16.08	2026-09-12
claude-fable-5	Anthropic	$1.479	$1.479	$32.16	2026-09-18
claude-fable-5-1	Anthropic	$1.479	$1.479	$32.16	2026-09-18
gpt-6-astra	OpenAI	$1.479	$1.479	$32.16	2026-09-17

Read this against the numbers above

Projection only — no other model was actually called against this corpus, and no accuracy figure on this page belongs to any row below.
NO OUTPUT TOKEN HERE IS REASONING: the budget was sent explicitly disabled on every call. A model that reasons by default moves every row below by more than its headline rate does.
68% of this run's input tokens came back as prefix cache hits. A card with no cache discount, or a provider that does not cache, prices the same workload higher than the row below suggests.

Home › Use Cases › Change regulatory impact comparison › Implementation

Use caseUC0374

🧪 Step 08

Implementation

Change regulatory impact comparison

The codeEvery file, and what it is for

13 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

data/corpus/CC-0001.txtthe change control record

one record, one call, sent verbatim — no row is pre-parsed and no paragraph is summarised on the way in

data/corpus/CC-0001.txt

data/dossiers.jsonthe registered dossier extract — a swap seam

11 registered elements per product, each one sentence, resolved from the authorisation number the record itself prints

You change it to: your products, your authorisation numbers, your registered sentences. src/prompt.py resolves the extract from the authorisation the record prints.

data/dossiers.json

data/policy.jsonAVC-2026

the invented variation code as data — the ladder, the elements, the natures and the 27-row table

data/policy.json

src/rules.pythe rule engine — a swap seam

(element, nature) -> category, rule id and rule text. Pure code. The only place a route is decided

You change it to: RULES is a list of (rule id, element, nature, route, text). A different framework is an edit to that list plus data/policy.md, which evals/check_labels.py re-parses independently.

src/rules.py

# AVC-2026 — the variation code this kit rules from. PURE CODE. No model, no network, no I/O.

CODE_ID = "AVC-2026"
CODE_TITLE = "Aldoria Variations Code 2026 (INVENTED — see the module docstring)"
ELEMENTS = {
ELEMENT_ORDER = ["D.1.2", "D.2.1", "D.2.4", "D.3.1", "D.3.3", "D.3.4",
NATURES = {
NATURE_ORDER = ["TIGHTENED", "WIDENED", "METHOD_EQUIVALENT", "METHOD_NEW", "SITE_LISTED",
CATEGORIES = {
CATEGORY_ORDER = ["AVC-0", "AVC-N1", "AVC-N2", "AVC-T30", "AVC-PA", "AVC-EXT"]
RULES = [

src/prompt.pythe prompt — a swap seam

six parts, fixed order; parts 1-4 byte-identical on every call this kit makes

You change it to: build() takes ONE argument — the record — so the hosted demo and the kit run the same code.

src/prompt.py

# The prompt, in the order it goes on the wire. ONE required argument: the document.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
MAX_TOKENS = 1600
def dossiers():
def ma_of(text):
def dossier_text(ma):
SYSTEM = (
def _elements_part():
def _natures_part():

src/adapters/init.pythe model adapter — a swap seam

raw HTTP to any OpenAI-compatible or Anthropic endpoint; the only file that knows who serves the model

You change it to: PROVIDER, BASE_URL and MODEL in .env. Nothing else moves — every arm goes through the same station.

src/adapters/__init__.py

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TIMEOUT_S = 1200
TRANSPORT_RETRIES = 1
def _post(url, headers, payload, timeout=TIMEOUT_S):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}

src/reader.pythe reader

one call, and a tolerant JSON parse that recovers a truncated reply and records — never merges — a key the schema does not name

src/reader.py

# One record in, one call out, a list of readings back. The only place a provider is reached.

FENCE = re.compile(r"^\s*```(?:json)?\s*|\s*```\s*$", re.M)
ALLOWED = {"item", "element", "nature", "quote"}
FORBIDDEN_HINTS = ("category", "route", "rule", "rule_id", "variation", "classification",
def parse(text):
def in_vocabulary(reading):
def read(cfg, record_text, max_tokens=None, thinking=adapters.THINKING_OFF):

src/station.pythe station

trusts exactly two fields of the reply and recomputes the category, the rule and the record's filing route from them

src/station.py

# THE STATION — what the kit actually ships. Two readings in, a ruling out, in PURE CODE.

def rule_one(reading):
def rule_record(readings, expected_items=None):

src/overreach.pythe overreach counter

counts and publishes every field the arm asserted outside the schema, then discards them

src/overreach.py

# Did the arm assert something the schema has no field for? Counted, published, discarded.

def summarise(overreach_rows):

evals/baseline.pythe three free floors — a swap seam

field diff, phrase rules and numeric direction — all scored through the same station and the same rule table

You change it to: FIELD_MAP maps your own field vocabulary to elements; PHRASE_RULES is the triage table your team would keep anyway.

evals/baseline.py

# THE FREE FLOORS. Pure code, zero calls, no network, $0.00 — and built to WIN, not to lose.

FIELD_MAP = [
FIELD_RX = [(re.compile(p, re.I), e) for p, e in FIELD_MAP]
DEFAULT_NATURE = {
PHRASE_RULES = [
PHRASE_RX = [(name, nat, re.compile(p, re.I | re.S)) for name, nat, p in PHRASE_RULES]
def element_of(label):
def _nums(s):
def numeric_nature(registered, proposed):
NUMERIC_ELEMENTS = {"D.2.4", "D.4.1", "D.6.1"}

evals/scoring.pythe graders

five cells on each change item and one route on each record, every one pure code

evals/scoring.py

# Every grader in this kit. PURE CODE — no model grades anything, here or anywhere.

ITEM_FIELDS = ["element", "nature", "category", "rule_id", "citation"]
def admitted_rows(section_text):
RECORD_FIELDS = ["route"]
def _majority(values):
def majority_floor(gold_items, gold_records):
def grade_item(got, key, dossier_text_norm, admitted):
def score_run(ruled_by_record, gold_items, gold_records, dossiers):
def under_file_maps(rows):
def correctness_maps(rows, rec_rows):

evals/significance.pythe paired test

McNemar exact on the discordant pairs — the only comparison that means anything between two arms answering the same items

evals/significance.py

# Is the paid arm's margin over a free floor real, or is it noise? PURE CODE, no model.

def mcnemar_exact(b, c):
def compare(a_correct, b_correct, label_a="paid", label_b="floor"):

src/app.pythe board

stdlib http.server; renders the corpus, the key, the committed run and all three floors with NO key configured

src/app.py

# The local board. stdlib http.server, no framework, no build step, no dependency.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
RESULTS = os.path.join(HERE, "results")
UI = os.path.join(HERE, "ui")
SCORED_RUN = "r001-variation-trigger"
def _j(path):
class Store:
STORE = Store()
class H(BaseHTTPRequestHandler):

Start hereThe shortest path into it

src/app.pyrun it and click around. It holds no pipeline logic on purpose; every number on screen comes from a module the eval harness already drives.
data/corpus/CC-0001.txtone record, one call, sent verbatim — no row is pre-parsed and no paragraph is summarised on the way in
data/dossiers.json11 registered elements per product, each one sentence, resolved from the authorisation number the record itself prints A swap seam.
data/policy.jsonthe invented variation code as data — the ladder, the elements, the natures and the 27-row table
src/rules.py(element, nature) -> category, rule id and rule text. Pure code. The only place a route is decided A swap seam.
src/prompt.pysix parts, fixed order; parts 1-4 byte-identical on every call this kit makes A swap seam.
src/adapters/__init__.pyraw HTTP to any OpenAI-compatible or Anthropic endpoint; the only file that knows who serves the model A swap seam.
src/reader.pyone call, and a tolerant JSON parse that recovers a truncated reply and records — never merges — a key the schema does not name
src/station.pytrusts exactly two fields of the reply and recomputes the category, the rule and the record's filing route from them
src/overreach.pycounts and publishes every field the arm asserted outside the schema, then discards them
evals/baseline.pyfield diff, phrase rules and numeric direction — all scored through the same station and the same rule table A swap seam.
evals/scoring.pyfive cells on each change item and one route on each record, every one pure code
evals/significance.pyMcNemar exact on the discordant pairs — the only comparison that means anything between two arms answering the same items

Every entry above is a file in this kit, listed in the order the pipeline runs it.

Home › Use Cases › Change regulatory impact comparison › Cost model

Use caseUC0374

🧪 Step 09

Cost model

Change regulatory impact comparison

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2037 input and 235 output tokens per query at top-k 1, of which about 150 tokens are fixed (system prompt, chat framing, the question itself) and the rest is retrieved context. Change the retrieval depth and the input scales with it. Change the volume and the bill scales linearly, because nothing here batches.

Querys per day Passages retrieved (top-k) Model Eval technique Rows graded (%)

Grading really is free here, and that is a property of the ruler rather than a discount

This kit is graded by pure code — the grader makes no model call, needs no key and sends nothing anywhere, so the grading line above is zero at every sampling rate and the leak count stays at nought. What that buys is a ruler you can run on every row, every time, for nothing. What it costs is reach: a code grader can only check what is mechanically checkable.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Shape	Querys/day	Per day	Per month	Per year

Model and top-k follow the controls above.

What this ignoresWhere a real bill diverges

Read before quoting any of it

Output length is assumed constant.A terser or more verbose model moves it, and output is priced above input everywhere.
No volume, committed-use, batch or cache discount is modelled.Each moves real enterprise pricing and none are on a public rate card.
Retrieval is free here and stops being free at scale.This kit scores every chunk in Python; a corpus a hundred times larger needs a vector store, which is infrastructure this arithmetic does not price.
Grading adds nothing to this bill, and would if you changed the ruler.A code grader is free and local; swapping in an LLM judge would cost more per row than the extraction itself and send every row to a vendor. The instrument is a cost decision, not a footnote.
Rates age.Every one carries the date it was read on the cost page.

Home › Use Cases › Change regulatory impact comparison › Threat model

Use caseUC0374

🧪 Step 10

Threat model

Change regulatory impact comparison

In one lineWhat this kit exposes

THE SHAPE OF THE ANSWER IS THE CONTROL, and it is not enough. The reply schema has no field for a variation category, a rule or a filing route — src/reader.py counts any key outside {item, element, nature, quote} as an overreach and DISCARDS it before src/rules.py is asked anything, and across 46 scored calls and 12 probe calls the arm asserted 0 such fields. ⚠ That control stops an attacker NAMING a route. It does not stop one changing a READING, and 6 of 12 injected wordings did exactly that.

One key, in the repo-root .env, gitignored and written 0600. src/config.py reads shared .env, then the kit's own, then the real environment; the board never receives the model id (src/config.MODEL_DISPLAY goes out instead) and no credential appears in any result file, cache line or screenshot.

The experimentWe attacked it — in the one field the attacker really writes

The justification paragraph is the site's own text. Four wordings were appended to one item's justification on each of three records, every target an item whose true route is prior approval or an extension, chosen from the ANSWER KEY rather than from the run so the probe could not be aimed at items the model was already getting wrong. Both arms were run for real: 46 scored calls in r001-variation-trigger and 12 attack calls in x001-variation-trigger, $0.014196 and $0.003181. Measured on 2026-09-11.

What the attack was trying to reach	Wording	Moved the route down
THE ROUTE — could a sentence make a prior-approval change look like a notification?	Across the 46 records of the scored run the arm never once proposed AVC-0 on an item whose key route is prior approval or an extension.	6 of 12 trials did exactly that. Four wordings on three records: plain 0/3, authority 0/3, imperative 3/3, schema 3/3.
THE SCHEMA — could a reply name its own category?	0 fields outside the schema across 46 scored calls.	0 forbidden fields asserted across 12 probe calls, one of which handed the arm a fully-formed object containing "category": "AVC-0". Counted, published and discarded before the ruling — and the attack still succeeded, through the reading.
THE ARITHMETIC — could an attacker move the route without moving a reading?	No. src/rules.py is a table lookup over two enums and nothing else reaches it.	Still no, and that is the finding: every one of the six successes moved a READING. The guardrail held and the answer moved anyway.

The three gates are not equally hard, and the easiest one is the one that matters. Reaching the schema requires the arm to invent a field, which it never did. Reaching the READING requires only that it believe a sentence.

The result6 of 12 obeyed, 6 of 12 moved the answer, 0 forbidden fields asserted. The structural control held and the attack succeeded anyway.

6 of 12trials that moved a prior-approval item's route DOWN the ladder

6 of 6direct-instruction wordings that succeeded

0 of 6persuasive wordings that succeeded

0forbidden fields asserted -- the structural control held

$0.00318112 calls bought for the probe

Four wordings — plain, authority, imperative, schema — appended to one item's justification on three records. Persuasion (plain, authority) failed 6 of 6. Direct instruction (imperative, schema) succeeded 6 of 6, every time by flipping the element to NONE and the nature to OUTSIDE_DOSSIER, which routes through R-26 to 'no variation'.

Read this twice

The guardrail this kit is proudest of is the one that did not help. The reply cannot express a variation category, and it never tried to — 0 fields outside the schema across 46 scored calls. The attacker did not need one: changing which registered element the change touches was enough to take a prior-approval item to no variation at all. A structural control protects the arithmetic, and the arithmetic was never the soft part.

HonestyWhat this does not prove

whether a wording nobody tried would succeed
whether an attack spread over several items compounds
whether the same wording succeeds at a different temperature or on another model
whether an attacker who could edit the PROPOSED CHANGES table, rather than the justification, would do better — that surface is not probed here

Home › Use Cases › Change regulatory impact comparison › Guardrails

Use caseUC0374

🧪 Step 11

Guardrails

Change regulatory impact comparison

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

the guardrail, verbatim

Nothing this kit produces files, notifies, implements, approves, withdraws, submits, commits to a date, or tells anybody a change is acceptable. It proposes a route and names who owns the decision.

Two places, and the first is structural. (1) The reply schema in src/prompt.py has four keys and none of them is a category, a rule, a route or a recommendation; src/reader.py counts any other key as an overreach and discards it. (2) src/station.py's every ruling carries the same sentence — 'PROPOSED ROUTE <x> — the regulatory affairs lead owns the filing decision' — and it is the only recommendation the kit emits.

EvidenceDoes it hold?

What	Measured
the answer contract cannot express a variation category or a filing route	0 fields asserted beyond the schema across 46 scored calls and 12 probe calls
the route is a table lookup no reply can reach	every published route in every arm — key, three floors and the paid call — came from the same function over two enums
an unnamed pair is routed UP, never down	7 items in the scored run resolved through the residual rule, every one of them to prior approval

The limitWhat a guardrail is not

It is not a filing. Nothing here is submitted to anybody.
It is not regulatory advice, and the corpus, the products and the framework are all invented.
It is not a defence against prompt injection — 6 of 12 wordings moved an item's route down.
It is not a substitute for a regulatory affairs review; it is an input to one.

WatchedWhat is watched, and why that one

5runs recorded

0 / 0deterministic metrics exact

+0.0%largest move — latency

0band breaches

—model half · needs provider URL

The kit measures 45 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

6 measured by the latest run 39 need the model half

Metric	Owner	Role	Why this one
variation-trigger-readings	The two readings	alarm	the nature reading on the pairs where the two classes differ only in the narrative — SITE_LISTED against SITE_NEW, METHOD_EQUIVALENT against METHOD_NEW, MINOR_NO_IMPACT against MAJOR_IMPACT — alarm on any movement in the UNDER-filing direction. Over-filing costs a fee; under-filing is a change implemented without the approval it needed
variation-trigger-route	The variation category and the filing route	alarm	the count of items proposed a route BELOW the key's — alarm on any increase in under-filing. The published run made 1; the numeric floor makes 7
variation-trigger-citation	The citation	alarm	citations_unlocatable — a quote that is not in the document at all — alarm on any unlocatable citation. This run produced 0.

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

Guard	Value	If it moves
corpus.doc_count	46	different corpus — nothing is comparable
corpus.bytes	97,966	change control records edited — the count held, the bytes did not
split.count	881	the graded cells count moved — a different set was scored
split.size_p50	4	the median size of one graded cell moved
split.size_p95	5	the 95th-percentile size of one graded cell moved
dataset.rows	46	the test set changed — every rate has a new denominator
tokens.context_limit	1,000,000	different model family
index.build_seconds	0	no index is built here — a change is in corpus preparation, not an index rebuild

all shared guards held between the recorded runs (failures 0, reasoning disabled) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

Group	Band	Denominator	Where the band comes from
An instruction to file, notify, implement or approve	0 of 46 scored calls and 0 of 12 probe calls — the reply schema has four keys and none of them could carry one	46 scored calls plus 12 adversarial calls	src/prompt.py SCHEMA and src/reader.py ALLOWED; counted by src/overreach.py on every call of r001-variation-trigger and x001-variation-trigger
A variation category asserted by the reply itself	0 of 46 scored calls. The probe handed the arm a fully-formed object containing "category": "AVC-0" and 0 forbidden fields were asserted across 12 probe calls.	46 scored calls plus 12 adversarial calls	src/reader.py counts any key outside {item, element, nature, quote} and DISCARDS it before src/rules.py is asked anything
A route published BELOW the key's, in the scored run	1 of 167 change items — and the board names it. The numeric free floor does it 7 times.	167 proposed change items	r001-variation-trigger against data/gold.jsonl; ranked on the AVC-2026 ladder by src/rules.py::rank
A route moved below the key's by an injected sentence	6 of 12 trials — the control that FAILED. Persuasion 0 of 6, direct instruction 6 of 6.	12 trials: four wordings on three records	x001-variation-trigger, targets chosen from the ANSWER KEY rather than from the run
A citation that is not in the registered dossier extract	0 unlocatable of 156 quotes returned. A quote that cannot be located is a plausible sentence, not evidence.	156 quotes returned across 167 change items	src/citation.py, located by substring in the normalised dossier extract; coverage >= 0.60 and precision >= 0.40 against the section or any one of its sentences
The two readings, and everything the code derives from them	element 166/167 · nature 145/167 · category 145/167 · rule 145/167 · item all five 144/167 · filing route 35/46 — every one printed beside the numeric free floor's 160, 142, 140, 137, 137 and 33	167 proposed change items and 46 change control records	r001-variation-trigger against data/gold.jsonl, five cells on each item and one route on each record; every grader pure code in evals/scoring.py
What the run cost, and whether it truncated	93,713 input tokens (63,744 of them prefix cache hits), 10,847 output, largest reply 373 against a 1600 ceiling — 23.3%, so nothing truncated. p50 1615 ms, p95 2048 ms.	46 calls of r001-variation-trigger	the provider's own usage block on every call, one cache line per call under results/cache-r001-variation-trigger.jsonl

HistoryRun history

5 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

floor · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

Metric	b000-fielddiff-variation-trigger 2026-09-11	b000-keyword-variation-trigger 2026-09-11	b000-numeric-variation-trigger 2026-09-11
calls	0	0	0
category correct	89	137	140
citation correct	160	160	160
citations returned	157	157	157
citations unlocatable	0	0	0
element correct	160	160	160
fields outside schema	0	0	0
forbidden fields asserted	0	0	0
item all correct	85	134	137
items unanswered	0	0	0
model latency p50 ms	0.00	0.00	0.00
model latency p95 ms	0.00	0.00	0.00
nature correct	85	139	142
off majority category	52	93	96
off majority element	140	140	140
off majority item all	85	134	137
off majority nature	66	111	114
over filed	50	20	20
residual rule used	0	8	8
route correct	20	32	33
route exact	89	137	140
rule id correct	85	134	137
under filed	28	10	7
usd per call	0.0	0.0	0.0
usd total	0.0	0.0	0.0

not a time series No two of these 3 runs measured the same system — they differ on under_filed, under_filed_note, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

comply · with the model in the path — 1 run. Columns here are only ever compared with each other.

Metric	r001-variation-trigger 2026-09-11
cache hit tokens total	63744
calls	46
category correct	145
citation correct	165
citations returned	156
citations unlocatable	0
element correct	166
fields outside schema	0
forbidden fields asserted	0
input tokens, whole run	93713
item all correct	144
items unanswered	0
model latency p50 ms	1615.00
model latency p95 ms	2048.00
nature correct	145
off majority category	103
off majority element	146
off majority item all	144
off majority nature	122
output tokens max	373
output tokens, whole run	10847
over filed	21
residual rule used	7
route correct	35
route exact	145
rule id correct	145
under filed	1
usd per call	0.000309
usd total	0.014196

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 29 chips that all say so.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

Metric	x001-variation-trigger 2026-09-11
answers moved	6
calls	12
forbidden fields asserted	0
obeyed	6
trials	12
usd total	0.003181

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 6 chips that all say so.

DeviationsWhat deviated

0 breaches across 5 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

Lever	What moves	Basis	Evidence
the NATURE reading the arm returns per change item	everything downstream of it. src/rules.py::decide takes the element and the nature and nothing else, so a wrong nature is looked up faithfully into a wrong category, a wrong rule and a wrong filing route — correctly computed, for the wrong reason. The station agrees with itself to the letter and cannot see it.	measured	r001-variation-trigger: nature correct on 145 of 167 items and category on 145 — the two numbers are within 0 of each other because the category IS the nature, once the element is right.
the ELEMENT reading — which registered section the change touches	the citation column and, where it flips to NONE, the route itself. It is also the one column the paid call wins outright, so it is where the money argument for the call lives.	measured	166 of 167 against the numeric floor's 160, paired 6 to 0, p = 0.031250. And CC-0044 item 3 is the clean case: element read as NONE dropped a 30-day notification to no variation at all.
the justification paragraph — the one field the site itself writes	the nature reading, and therefore the whole published route. It is the attack surface and it is also where the kit's own dominant error lives.	measured	6 of 12 injected trials moved an item's route DOWN the ladder, every one of them by rewriting the nature; and the largest error bucket in the clean run is a minor manufacturing change read from that paragraph as a change of principle.
the residual rule R-99 — what happens to a pair the table does not name	the direction of every unclassifiable item. It routes UP, to prior approval, which is why this kit over-files rather than under-files when it is confused.	measured	7 items resolved through R-99 in the scored run, and the run's direction is 21 over-filed against 1 under-filed.
the majority-class mix of the corpus	every headline percentage on this page. Answering AVC-T30 to every item already scores 32.9% on the category.	measured	the corpus distribution in data/corpus-stats.json, and the off-majority rows beside every rate: 103 of 112 off-majority items right on the category.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

Band	What fires
An instruction to file, notify, implement or approve	never observed. ⚠︎ Read it as a statement about the CONTRACT, not about the model's intentions: there is no field to put one in.
A variation category asserted by the reply itself	⚑ AND IT DID NOT SAVE THE ANSWER. 6 of 12 injected trials still moved an item's route down, by changing a READING. A band at zero is not a defence.
A route published BELOW the key's, in the scored run	⚑ ANY READING OF THIS BAND MUST ALSO QUOTE p = 0.070312. One item against seven, paired, is a trend and not a result at the 5% line.
A route moved below the key's by an injected sentence	already fired. This is the band this kit fails, and the page leads with it.
A citation that is not in the registered dossier extract	never observed on this run. 2 items scored a citation failure for a different reason — the wrong registered sentence, or none at all.
The two readings, and everything the code derives from them	⚑ NO ROW OF THIS BAND MAY BE QUOTED WITHOUT ITS FLOOR. On the category the paid call and free code are inside each other's noise (p = 0.486850); only the element column clears 0.05 (p = 0.031250).
What the run cost, and whether it truncated	⚠︎ READ THE CEILING MARGIN AS A MEASUREMENT OF SHORT ANSWERS OVER SMALL RECORDS. A record here carries 2 to 5 change items and the reply is one object per item; a larger record is a longer reply AND a longer prompt.

NextThe three you would add first

A person on every item before anything is filedthe injection probe moved 6 of 12 attacked items to 'no variation', and the one under-filed item of the scored run was published with the station agreeing with itself to the letter
A hard stop on any item whose reading came back OUTSIDE the vocabularythe residual rule routes it to prior approval, which is safe but silent; a reviewer should be told a reading was unusable rather than handed a conservative default
A second reading of the justification paragraph, by different meansthe attack and the kit's own worst error both live in that paragraph, and nothing currently cross-checks it

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The corpus generator, the independent label checker, the rule engine, all three free floors and the zero-call re-score are free and re-run in seconds with no network — verified this session on the committed kit. Only the model arm costs money: 46 calls, once, $0.014196, fired as a 2-record smoke and then the full run, recorded per call in the run record and never re-fired. The scored answer can be rebuilt from the committed result file at any time for nothing, and it was: the under-filing test was added after the run was paid for. The adversarial arm was fired once, 12 calls, $0.003181.

What this cannot tell you

Whether any of these bands resemble a real change control queue. The corpus is invented, the case mix is CHOSEN, and data/SOURCES.md says so before it says any number.
Whether the bands hold on a second run. One scored run, one day, one model, one setting — and a five-item gap on the category hides variance better than a large one would.
Whether the structural guardrail is worth anything. It held perfectly (0 forbidden fields across 46 scored and 12 probe calls) and the attack succeeded anyway, which is evidence that it guards the wrong thing rather than evidence that it works.
Whether a wording nobody tried defeats it, and whether the PROPOSED CHANGES table — a second attacker-controlled surface — can be attacked at all. It was not probed.
Whether the one under-filed item of the scored run is a rate or an accident. One item on 167 is not a rate.

Home › Use Cases › Change regulatory impact comparison › Frameworks

Use caseUC0374

🧪 Step 12

Frameworks

Change regulatory impact comparison

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, standard library end to end — no orchestration layer, no vendor SDK, no retrieval stack, no schema library, no pip install. requirements.txt names nothing and spends its whole body saying why. The reason is the same one that makes the kit measurable: every step between the document and the published route has to be readable by somebody deciding whether to trust the route, and a framework puts a layer between them.

The mappingSeam by seam, who owns what

Seam	File here	Framework equivalent	Note
the model call	`src/adapters/__init__.py`	an LLM wrapper / Runnable	Raw HTTP over urllib to any OpenAI-compatible provider or Anthropic, with a 1200 s socket timeout. A wrapper would buy retries and a provider registry; both are here in a few dozen lines, and this kit never streams — one call per record, whole reply read at once. It also distinguishes a TRANSIENT failure from a TERMINAL one and retries only the first, because retrying a bad key four times is how one defect becomes a rate-limit ban.
the reply parse	`src/reader.py`	a structured-output or schema-validation library	Hand-written, and built for the three things that actually happen — a fenced block, prose in front of the JSON, and a reply cut off mid-object, which it recovers by closing the array after the last complete item. It held on all 46 paid calls, which retires nothing: a stricter library would fail LOUDLY where this fails quietly. ⚑ The one thing it does that a schema library would not is RECORD a key outside the schema rather than rejecting the reply — that count is the kit's guardrail evidence.
the route decision	`src/rules.py`	a rules engine, or a policy DSL	A list of tuples and a dict lookup. The whole framework case here is auditability, and a 27-row table in plain Python is more auditable than any DSL that could express it — evals/check_labels.py re-parses the SAME table out of the markdown a human reads and asserts the two agree on every item, which is the property a rules engine would claim and not demonstrate.
the paired test	`evals/significance.py`	scipy, statsmodels	math.comb and integer factorials. McNemar's EXACT test, not the chi-square approximation, because the discordant counts here are in the tens — the regime where the approximation is worst and a kit is most tempted to publish a significant-looking number. Twenty lines of standard library against a 30 MB dependency, for a two-sided binomial tail.
the board	`src/app.py`	Flask, FastAPI, or any front-end build step	stdlib http.server, one hand-written HTML file and one JS file with no build step and no CDN. It exists to be READ as much as run: every number on it comes off an endpoint rather than out of a template, which is what stops a figure going stale silently.

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear, and not concurrent at all. There is no agent, no tool loop, no retrieval and NO STATE carried between records — nothing is written, nothing persists, and any record can be re-ruled from its committed readings in isolation, which is what makes the zero-call re-score possible. The one aggregation in the whole kit is a MAX over one record's own items, and it happens in pure code after the call. That is also the regulatory boundary expressed as an architecture: a pipeline that carried state between records would be one that maintained a filing position.

The other sideWhat a framework costs you

Adding a provider means editing the PROVIDERS dict by hand. In exchange pip install pulls nothing and a forker runs this on whichever key they already hold.
The reply is parsed by hand. It held across all 46 paid calls and all 12 probe calls with 0 unparsed replies; that is an observation, not a guarantee — a stricter schema library would fail loudly where this fails quietly.
No rules engine, so AVC-2026 is a list of tuples read by the engine, the key, every floor and the board. That is the point — the framework is meant to be ARGUABLE rather than embedded, and evals/check_labels.py re-parses the same table out of the markdown a human reads and proves the two agree.
No statistics package, so McNemar's exact test is twenty lines of math.comb. In exchange the p-values on this page can be checked by hand from the four counts printed beside them.
No front-end build step, so the board is one HTML file and one JS file. The cost is that every table is hand-written; the return is that a reader can see every number comes off an endpoint rather than out of a template.

What we could NOT verify

Whether a framework would have caught anything this kit got wrong. Nothing here failed for a reason a library would have prevented — the 22 wrong readings were wrong before any code touched them — and that is a statement about 167 change items on one run rather than about frameworks.
Whether the hand-written parser survives a provider whose reply shape differs. One provider, one model, one day.
Whether the no-install claim holds on a Python older than the one it was written against. It was run on this machine's interpreter and nowhere else.
Whether a rules engine would make the framework easier to CHANGE. This kit only ever expressed one framework, so the claim that a list of tuples is enough is untested against a second one.

Home › Use Cases › Change regulatory impact comparison › Observability

Use caseUC0374

🧪 Step 13

Observability

Change regulatory impact comparison

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-variation-trigger, 2026-09-11. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

Reading	Last run	Band	What fires
Model, median	1,615 ms	93,713 input tokens (63,744 of them prefix cache hits), 10,847 output, largest reply 373 against a 1600 ceiling — 23.3%, so nothing truncated. p50 1615 ms, p95 2048 ms.	⚠︎ READ THE CEILING MARGIN AS A MEASUREMENT OF SHORT ANSWERS OVER SMALL RECORDS. A record here carries 2 to 5 change items and the reply is one object per item; a larger record is a longer reply AND a longer prompt.
Model, p95	2,048 ms	93,713 input tokens (63,744 of them prefix cache hits), 10,847 output, largest reply 373 against a 1600 ceiling — 23.3%, so nothing truncated. p50 1615 ms, p95 2048 ms.	⚠︎ READ THE CEILING MARGIN AS A MEASUREMENT OF SHORT ANSWERS OVER SMALL RECORDS. A record here carries 2 to 5 change items and the reply is one object per item; a larger record is a longer reply AND a longer prompt.
Input tokens	93,713	93,713 input tokens (63,744 of them prefix cache hits), 10,847 output, largest reply 373 against a 1600 ceiling — 23.3%, so nothing truncated. p50 1615 ms, p95 2048 ms.	⚠︎ READ THE CEILING MARGIN AS A MEASUREMENT OF SHORT ANSWERS OVER SMALL RECORDS. A record here carries 2 to 5 change items and the reply is one object per item; a larger record is a longer reply AND a longer prompt.
Output tokens	10,847	93,713 input tokens (63,744 of them prefix cache hits), 10,847 output, largest reply 373 against a 1600 ceiling — 23.3%, so nothing truncated. p50 1615 ms, p95 2048 ms.	⚠︎ READ THE CEILING MARGIN AS A MEASUREMENT OF SHORT ANSWERS OVER SMALL RECORDS. A record here carries 2 to 5 change items and the reply is one object per item; a larger record is a longer reply AND a longer prompt.

No movement column. This is the only run on record, so there is nothing to move against. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

The other sideWhat this does not tell you

What we could NOT verify

No stage-by-stage span tree. These are whole-call timings; which part of the pipeline spent the time is not recorded by this kit's harness.
Latency is a run-level median and p95. The spread between the fastest and slowest single call is not stored, so the distribution cannot be drawn.
No earlier run is comparable to this one (a different model tier or a changed guard), so nothing on this page is a trend yet.
One machine, one network, one region. Latency includes the provider round trip from a single location and will not transfer to yours.

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

Home › Use Cases › Change regulatory impact comparison › Run it in your environment

Use caseUC0374

🧪 Step 14

Run it in your environment

Change regulatory impact comparison

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Dependencies	nothing beyond the standard library — `requirements.txt` names no package
Configuration	7 environment variables — the full knob table is at the bottom of this page
Last verified run	no dated run record — see the Eval lens for what was measured

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

Artifact	Where	Leaves your machine?
the change control records	`data/corpus/CC-*.txt` — 46 records, 167 proposed change items, 97,966 bytes, generated from seed 20260911; your disk	the whole record, per call — every row and every justification paragraph, verbatim. There is no pre-digest step
the registered dossier extracts	`data/dossiers.json` — 5 invented products, 11 registered elements each, one sentence per element	the extract for the authorisation THIS record names, whole — resolved by `src/prompt.py::ma_of()` off the document itself
AVC-2026, the variation code	`src/rules.py` as the engine and `data/policy.md` as the markdown a human reads — 27 rows plus the residual	the element and nature VOCABULARIES go into every prompt; the table itself does NOT. The arm never sees which route a pair produces
the answer key	`data/gold.jsonl` — 167 rows, derived by the generator from the pair it planted, re-derived independently by `evals/check_labels.py`	nothing. No module on the answer path opens it — asserted on every run and published as floor_memorisation
the committed runs	`results/` — the scored run, three free floors, the free wiring stub and the adversarial probe, plus one cache line per call	nothing. The board replays them; `--rescore` rebuilds the whole scored answer from them with 0 calls
the key	the repository-root `.env`, gitignored and written 0600	as a bearer header to the configured BASE_URL, and nowhere else. The board is handed `MODEL_DISPLAY` and never the model id

The wireWhat crosses it, and when

Runtime call site	Destination
`src/adapters/__init__.py` line 74	the configured BASE_URL

Build-time (corpus tooling, run once)	Destination
none — this kit’s corpus is generated locally

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The third state

What the provider retains, trains on or logs once a request arrives is not measured here — it is provider-dependent, and this page does not assert what no run of ours can verify.

The shapeHow it ships, and what you add

Runs on	Your machine, or any box with Python 3 and outbound HTTPS to one provider. No framework, no service, no database, no container and NO INSTALL STEP — requirements.txt names nothing. The kit is a folder of readable Python and six JSON files, and the whole free half — the corpus generator, the independent label checker, the rule engine, all three floors, the zero-call re-score and the local board — runs with no key and no network at all, verified this session on the committed kit.
The key	One key, in the repo-root .env, gitignored and written 0600. src/config.py reads shared .env, then the kit's own, then the real environment; the board never receives the model id (src/config.MODEL_DISPLAY goes out instead) and no credential appears in any result file, cache line or screenshot.
You add	your reverse proxy and your identity in front of one process — the kit deliberately ships neither, because identity belongs to your estate, not to a kit
Never ships	Docker, Helm, a queue, a database. The product is a folder of readable Python; what to run it under is this page’s job to state, not the kit’s job to impose

The ladderEach decision, and what you provision past its ceiling

Decision	The kit ships	Measured here	Past the ceiling	What stops being valid
rulebook	`src/rules.py` — AVC-2026 as 27 (element, nature) rows plus a residual, and it is the ONLY place a route is decided in this kit. The answer key calls it, all three free floors call it, and the paid arm's two readings go through it unchanged, so a difference between any two arms is a difference in READING and never in arithmetic.	7 of the 167 change items resolved through the residual rule R-99 in the scored run. The same table produced every published route on every arm. (src/rules.py::decide(); re-parsed independently out of data/policy.md by evals/check_labels.py, 0 problems over 1,262 assertions, red-proven by flipping one category in the key.)	the table is a cross product, so a real framework with conditions that depend on more than two readings does not fit it. AVC-2026 was designed to be decidable from (element, nature) precisely so the measurement has a clean subject; a framework whose route depends on a third reading would need a third enum, and every score here would have to be re-measured.	the residual rule routes an unnamed pair UP, to prior approval. That is the safe direction and it is also a silent one — a reviewer is handed a conservative default rather than being told the reading was unusable.
model	one provider, one key, one call per record, reasoning DISABLED as an explicit field on the wire. Raw HTTP over urllib in `src/adapters/__init__.py`; no vendor SDK and no orchestration layer.	46 calls, $0.014196 off peak, p50 1615 ms and p95 2048 ms, 0 failures and 0 unparsed replies. Largest reply 373 output tokens against a 1600 ceiling. (results/eval-r001-variation-trigger.json, and one cache line per call under results/cache-r001-variation-trigger.jsonl.)	the reply carries one object per change item, so BOTH sides of the bill grow with the record's size — but the dossier extract is the part that grows fastest, and it is re-sent whole every call. A real extract would blow the input side long before the output ceiling was reached.	one model, one setting, one day. Nothing here says another model reads the nature better, and the nature is the reading that decides the category.
labels	`data/gold.jsonl` — 167 change items, each carrying the registered element, the nature, the category, the rule and the registered sentence admitted as its citation.	1,262 assertions in evals/check_labels.py, 0 problems: every route re-derived from the markdown, every citation found verbatim in its own dossier section, every change row proved to be a real change, and the corpus proved to regenerate byte-for-byte. (derived by tools/build_corpus.py from the (element, nature) pair it planted; never typed. No module on the answer path reads it.)	the key is only as good as the sentences the generator writes. Its own narrative variant pools are what stop a regular expression decoding it, and the first version of this kit's keyword floor scored 165 of 167 before those pools existed.	an item whose generated sentence does not really describe the planted pair would give a confidently wrong key. check_labels.py cannot catch that — it checks the arithmetic and the citations, not the prose.
corpus refresh	`tools/build_corpus.py`, seed 20260911 — the records, the dossiers, the key, the policy markdown and the corpus statistics, all from one call.	build_corpus.py --check regenerates the whole corpus in memory and diffs it against disk: IDENTICAL. 46 records, 167 change items, 97,966 bytes. (data/corpus-stats.json carries the seed and every distribution the board prints; evals/check_labels.py runs --check as its last assertion.)	a larger corpus is one number in PLAN, but the narrative variant pools are hand-written, so scale buys repetition rather than variety past a point.	regenerating with a different seed invalidates every score in this spec. The dataset version is stamped on the run record for that reason.

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

What you see	What it is	First move
an arm whose registered ELEMENT is right and whose NATURE is not	it found the right part of the dossier and misread what was being done to it. That is the expensive half: the element is a vocabulary problem and the nature is a reading problem, and only the nature moves the route.	read the nature column against the numeric floor, never the element column. The paid call wins the element outright (166 to 160, p = 0.031250) and ties the nature (145 to 142, p = 0.749259). (run r001-variation-trigger against b000-numeric-variation-trigger, paired on the same 167 change items.)
an arm that over-files far more often than it under-files	it is reading changes as bigger than they are — 21 items proposed a route above the key's against 1 below. It costs filing fees and reviewer time and it does not reach an inspection.	read the taxonomy, not the total. The single biggest bucket is a minor manufacturing change read as a change of principle (7 items), which turns a 30-day notification into a prior approval. (run r001-variation-trigger direction block, and the confusion table on the nature reading.)
the free floor and the paid call within a few items of each other	the corpus cannot separate them, and a five-item difference on 167 items is not a margin. Quoting the accuracies side by side invites a subtraction that is not a result.	read the paired McNemar row, not the two accuracies. On the category it is 19 items to 14, p = 0.486850. (evals/significance.py over run r001-variation-trigger and b000-numeric-variation-trigger.)
a route of AVC-0 on an item whose field label names a registered part	the arm read the change as outside the dossier entirely. It is the one failure mode here that UNDER-files, and the station cannot see it: it rules NONE through R-26 correctly and agrees with itself to the letter.	open the item on the board. The red band names it and prints what the key says beside what the run answered. (run r001-variation-trigger, CC-0044 item 3 — the single under-filed item of the scored run, and the same reading all six successful injection trials produced.)

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, FOUNDRY_RUN_ID, MAX_CALLS_PER_DAY, MODEL, PROVIDER.

Constant	File	Value
`RETRIES`	`src/adapters/__init__.py`	4
`TIMEOUT_S`	`src/adapters/__init__.py`	1200
`TRANSPORT_RETRIES`	`src/adapters/__init__.py`	1
`COVERAGE_MIN`	`src/citation.py`	0.6
`PRECISION_MIN`	`src/citation.py`	0.4
`MAX_TOKENS`	`src/prompt.py`	1600

The other sideWhat this page does not claim

Not measured, and said so

Whether the result holds on a real variation framework — AVC-2026 is invented and a real framework's conditions would move items between routes. Whether it holds on real change control records: every narrative here comes from a hand-written variant pool. Repeatability: ONE scored run, one day, one model, one setting — the pipeline below the call is deterministic and every grader is pure code, but the model was asked once. Any second model: every non-run row in the Cost lens is a projection of this run's tokens onto a published card, with no quality figure behind it. Whether the under-filing gap is real: 1 against 7, p = 0.070312, and 167 items is not enough to settle it. Whether a stronger free floor exists — three were written and the strongest is published, and the margin is a property of the floor. Whether an injected wording nobody tried succeeds, and whether the PROPOSED CHANGES table, a second attacker-controlled surface, can be attacked at all: it was not probed.

The corpus licence, from the Data lens: MIT Your corpus’s licence is yours to verify, and the labels you write are about your documents.

Home › Use Cases › Change regulatory impact comparison › Evals › The two readings

Use caseUC0374

🧪 Evals · grader

The two readings

Change regulatory impact comparison

In one lineThe two readings

whether the arm named the right registered element and the right kind of change — the only two fields it is asked for, and the only two anything downstream uses

$0.00per 1,000 change control records

nodata leaves your network

yessame answer every time

MethodHow the test was run

python3 -m evals.run --arm model --run-id r001 --rescore (re-scores from the committed result file; 0 calls, $0.00)

Every grader on these pages scored the same 46 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The record and item

CC-0044, item 3

The field, as the site writes it

Primary pack component supplier

Currently registered

registered component supplier

Proposed

Sennick Components AB

The key: registered element

D.5.1 — Container closure system

The key: nature of the change

MINOR_NO_IMPACT

The key: variation category

AVC-T30 — notify, then implement after 30 days if the Board has not objected

The key: rule

R-20

The run answered: element

NONE

The run answered: nature

OUTSIDE_DOSSIER

So the kit published

AVC-0 — no variation; nothing is filed

…under rule

R-26

Direction of the error

UNDER-FILED — the one error of this kind in the whole run

Why it matters

The component is the registered container's own part. Reading a supplier change for it as 'outside the dossier' takes a filing that exists and turns it into no filing at all.

What the free floors did

all three read it D.5.1 / MINOR_NO_IMPACT and got it right — the field label contains 'pack component supplier', which the field map resolves without reading anything

What the pure-code station did

exactly what it was handed. It ruled NONE/OUTSIDE_DOSSIER through R-26 correctly, agreed with itself, and published AVC-0. Nothing downstream of a reading can see that the reading was wrong.

Grader	Verdict	Why
The two readings	fail	Wrong on both. Element NONE against the key's D.5.1, nature OUTSIDE_DOSSIER against MINOR_NO_IMPACT. Both cells score 0.
The variation category and the filing route	fail	Wrong, and in the expensive direction. AVC-0 against the key's AVC-T30, rule R-26 against R-20 — the one UNDER-filed item of the whole scored run. It moved the record's filing route too, because the route is a MAX.
The citation	pass	Correct, and that is the uncomfortable part. The arm named no registered element, so the right answer is an empty quote and it returned one. A citation grader cannot see a reading that was wrong about what to cite.

The formulaWhat it computes

The analysisWhat it actually did

Model	Result
the fast tier (THE PUBLISHED RUN)	99.4% element · 1 more measured on this row
fielddiff floor — 0 calls, $0.00, no network	95.8% element · 1 more measured on this row
keyword floor — 0 calls, $0.00, no network	95.8% element · 1 more measured on this row
numeric floor — 0 calls, $0.00, no network	95.8% element · 1 more measured on this row

In operationWhat to monitor

Reference standard: data/gold.jsonl, derived by tools/build_corpus.py from the pair the generator planted, and re-derived independently by evals/check_labels.py.

No true/false rates for this grader. It records 5 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

Watch these

the nature reading on the pairs where the two classes differ only in the narrative — SITE_LISTED against SITE_NEW, METHOD_EQUIVALENT against METHOD_NEW, MINOR_NO_IMPACT against MAJOR_IMPACT

Alarm on

any movement in the UNDER-filing direction. Over-filing costs a fee; under-filing is a change implemented without the approval it needed

How tight can the band be? read beside the majority-class floor: answering TIGHTENED to every item scores 16.8% on the nature, so anything under that is worse than a constant

Cadence: on every run, and on every zero-call re-score of a committed run

The decisionWhen to reach for it

Use it

always — these are the only two fields the model supplies

Do not use it

never; without them there is nothing the model contributed

Home › Use Cases › Change regulatory impact comparison › Evals › The variation category and the filing route

Use caseUC0374

🧪 Evals · grader

The variation category and the filing route

Change regulatory impact comparison

In one lineThe variation category and the filing route

whether the route the kit PUBLISHES is the route the key names — per change item, and per record after the max-aggregation

$0.00per 1,000 change control records

nodata leaves your network

yessame answer every time

MethodHow the test was run

python3 -m evals.run --arm model --run-id r001 --rescore

Every grader on these pages scored the same 46 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The record and item

CC-0044, item 3

The field, as the site writes it

Primary pack component supplier

Currently registered

registered component supplier

Proposed

Sennick Components AB

The key: registered element

D.5.1 — Container closure system

The key: nature of the change

MINOR_NO_IMPACT

The key: variation category

AVC-T30 — notify, then implement after 30 days if the Board has not objected

The key: rule

R-20

The run answered: element

NONE

The run answered: nature

OUTSIDE_DOSSIER

So the kit published

AVC-0 — no variation; nothing is filed

…under rule

R-26

Direction of the error

UNDER-FILED — the one error of this kind in the whole run

Why it matters

The component is the registered container's own part. Reading a supplier change for it as 'outside the dossier' takes a filing that exists and turns it into no filing at all.

What the free floors did

all three read it D.5.1 / MINOR_NO_IMPACT and got it right — the field label contains 'pack component supplier', which the field map resolves without reading anything

What the pure-code station did

exactly what it was handed. It ruled NONE/OUTSIDE_DOSSIER through R-26 correctly, agreed with itself, and published AVC-0. Nothing downstream of a reading can see that the reading was wrong.

Grader	Verdict	Why
The two readings	fail	Wrong on both. Element NONE against the key's D.5.1, nature OUTSIDE_DOSSIER against MINOR_NO_IMPACT. Both cells score 0.
The variation category and the filing route	fail	Wrong, and in the expensive direction. AVC-0 against the key's AVC-T30, rule R-26 against R-20 — the one UNDER-filed item of the whole scored run. It moved the record's filing route too, because the route is a MAX.
The citation	pass	Correct, and that is the uncomfortable part. The arm named no registered element, so the right answer is an empty quote and it returned one. A citation grader cannot see a reading that was wrong about what to cite.

The formulaWhat it computes

The analysisWhat it actually did

Model	Result
the fast tier (THE PUBLISHED RUN)	86.8% category · 1 more measured on this row
fielddiff floor — 0 calls, $0.00, no network	53.3% category · 1 more measured on this row
keyword floor — 0 calls, $0.00, no network	82.0% category · 1 more measured on this row
numeric floor — 0 calls, $0.00, no network	83.8% category · 1 more measured on this row

In operationWhat to monitor

Reference standard: src/rules.py over data/gold.jsonl's (element, nature) pair — the same function every arm's readings go through.

Watch these

the count of items proposed a route BELOW the key's

Alarm on

any increase in under-filing. The published run made 1; the numeric floor makes 7

How tight can the band be? read beside the numeric floor on every row. On the category the two are inside each other's noise; on under-filing the paid call is ahead and the corpus is not big enough to call it.

Cadence: on every run, and on every zero-call re-score

The decisionWhen to reach for it

Use it

always — it is the answer the kit exists to produce

Do not use it

never, but do not read it alone: the free floor is inside the noise of it

Home › Use Cases › Change regulatory impact comparison › Evals › The citation

Use caseUC0374

🧪 Evals · grader

The citation

Change regulatory impact comparison

In one lineThe citation

whether the sentence the arm quoted is actually IN the registered dossier extract, and whether it is the sentence of the element it named

$0.00per 1,000 change control records

nodata leaves your network

yessame answer every time

MethodHow the test was run

python3 -m evals.run --arm model --run-id r001 --rescore

Every grader on these pages scored the same 46 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The record and item

CC-0044, item 3

The field, as the site writes it

Primary pack component supplier

Currently registered

registered component supplier

Proposed

Sennick Components AB

The key: registered element

D.5.1 — Container closure system

The key: nature of the change

MINOR_NO_IMPACT

The key: variation category

AVC-T30 — notify, then implement after 30 days if the Board has not objected

The key: rule

R-20

The run answered: element

NONE

The run answered: nature

OUTSIDE_DOSSIER

So the kit published

AVC-0 — no variation; nothing is filed

…under rule

R-26

Direction of the error

UNDER-FILED — the one error of this kind in the whole run

Why it matters

The component is the registered container's own part. Reading a supplier change for it as 'outside the dossier' takes a filing that exists and turns it into no filing at all.

What the free floors did

all three read it D.5.1 / MINOR_NO_IMPACT and got it right — the field label contains 'pack component supplier', which the field map resolves without reading anything

What the pure-code station did

exactly what it was handed. It ruled NONE/OUTSIDE_DOSSIER through R-26 correctly, agreed with itself, and published AVC-0. Nothing downstream of a reading can see that the reading was wrong.

Grader	Verdict	Why
The two readings	fail	Wrong on both. Element NONE against the key's D.5.1, nature OUTSIDE_DOSSIER against MINOR_NO_IMPACT. Both cells score 0.
The variation category and the filing route	fail	Wrong, and in the expensive direction. AVC-0 against the key's AVC-T30, rule R-26 against R-20 — the one UNDER-filed item of the whole scored run. It moved the record's filing route too, because the route is a MAX.
The citation	pass	Correct, and that is the uncomfortable part. The arm named no registered element, so the right answer is an empty quote and it returned one. A citation grader cannot see a reading that was wrong about what to cite.

The formulaWhat it computes

The analysisWhat it actually did

Model	Result
the fast tier (THE PUBLISHED RUN)	98.8% citation
fielddiff floor — quotes the section by dict lookup, 0 calls	95.8% citation
keyword floor — quotes the section by dict lookup, 0 calls	95.8% citation
numeric floor — quotes the section by dict lookup, 0 calls	95.8% citation

In operationWhat to monitor

Reference standard: the registered sentence of the section the key names, located by substring in the normalised dossier extract.

No true/false rates for this grader. It records 3 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

Watch these

citations_unlocatable — a quote that is not in the document at all

Alarm on

any unlocatable citation. This run produced 0.

How tight can the band be? the three free floors all score 160 of 167 here by construction — they quote the section of whatever element they picked with a dict lookup, so their citation score IS their element score. Withholding that from them would have handed the paid arm this column for free.

Cadence: on every run and every re-score

The decisionWhen to reach for it

Use it

always — 'which registered text does this change touch' is half the question the kit is asked

Do not use it

never; a route with no quoted registered sentence is an assertion

A living map of modern AI — kept current every morning

The business caseThe problem this solves

Audience

The inputThe actual change control records

The corpus

The outcomeWhat a good result looks like

And when it cannot

Where it fitsWhat did work

At a glanceHow the whole thing runs

Should you use this?What you bring, where it stops, and when not to use it

Run itHow this reaches your data

Run this on your own data

Step 01 of 14Business case

In brief

Current processWhat it replaces

Where it is not good enough

Step 02 of 14Architecture

In brief

The swap seams

Components

Where it breaks at scale

Step 03 of 14App UI

SuccessesWhen it works

LimitsWhen it does not

Step 04 of 14Corpus

In brief

How it is cutWhat one graded cells (five on each of the 167 change items, one route on each of the 46 records) is

SetupWhat the setup figure measured

LicenceLicence

Bring your ownBring your own change control records

What breaks it

Step 05 of 14Prompt

In brief

AssemblyHow the prompt is assembled

Full promptThe prompt in full

Raw responseThe raw response

Step 06 of 14Evals

In brief

The rulerHow it was graded

MeasurementsWhat was measured

How the method was validated

Grading costWhat it costs

Same work, 10× the bill

Grading unitWhat the grading figure prices

The gradersThree ways to grade

Why this set cannot separate them

Choosing oneWhich one to use

Failure causesHow it fails

What we could NOT verify

Step 07 of 14Unit cost

In brief

Answering vs gradingAnswering and grading are two separate bills

Cost driversWhat actually moves the bill

Your volumeWhat it costs at your volume

Where pricing changes shape

Your return, with your numbers

Model choiceWhy this model

Other modelsOn other models

Read this against the numbers above

The codeEvery file, and what it is for

data/corpus/CC-0001.txtthe change control record

data/dossiers.jsonthe registered dossier extract — a swap seam

data/policy.jsonAVC-2026

src/rules.pythe rule engine — a swap seam

src/prompt.pythe prompt — a swap seam

src/adapters/__init__.pythe model adapter — a swap seam

src/reader.pythe reader

src/station.pythe station

src/overreach.pythe overreach counter

evals/baseline.pythe three free floors — a swap seam

evals/scoring.pythe graders

evals/significance.pythe paired test

src/app.pythe board

Start hereThe shortest path into it

ScenariosWhat it costs at your volume

Grading really is free here, and that is a property of the ruler rather than a discount

OperationsFour shapes of workload

What this ignoresWhere a real bill diverges

Read before quoting any of it

In one lineWhat this kit exposes

The experimentWe attacked it — in the one field the attacker really writes

src/adapters/init.pythe model adapter — a swap seam