The business caseThe problem this solves

"Which wires are over the callback threshold" is a comparison, and the payment hub has been doing it for years, for nothing. The question a release desk cannot answer from a screen is the next one, and it has two halves. FIRST: was the beneficiary's account, IBAN, bank identifier or bank name CHANGED after the instruction arrived? That is raised once, in one window of the case file, and never restated -- three sweeps later the instruction reads exactly like an instruction nobody ever touched. SECOND: do the callbacks logged against it COUNT? A callback counts only when it was placed to a number the bank already held; a number the sender supplied is the same channel again, and confirming an instruction against itself confirms nothing. Both halves are prose typed by a payments clerk, and both of them decide whether the money moves. Somebody working down the release queue at each sweep: opening each unreleased wire, reading the change log to decide whether anything in it touched the four fields the policy names, reading the callback log to decide which of those calls actually counted, adding them to however many counted at the last look, and remembering whether this one has already gone to the second approver.

The inputThe actual release-queue snapshots

The corpus is 180 release-queue snapshots, 0.90 MB (json 2 · jsonl 1 · txt 180). AN OUTBOUND WIRE QUEUE IS A LIST OF WHO IS PAYING WHOM, HOW MUCH, AND THROUGH WHICH BANK, WITH THE CUSTOMER'S AUTHORISED SIGNATORIES AND THEIR DIRECT DIALS ATTACHED. There is no public one and there never will be: a bank that published one would have committed exactly the disclosure this kit's own privacy guard exists to prevent. The two alternatives were both worse. A SCRUBBED REAL QUEUE is the step that quietly changes what the model is being asked to read -- a change-log line with the account number taken out is not the line the clerk typed, and those lines are the entire judgement here. NOTHING AT ALL is a description of a kit. ⚑ AND THE SEED WAS CHOSEN, WHICH IS WORTH SAYING OUT LOUD. A hundred candidates were scored on ANSWER-CLASS BALANCE alone -- every disposition present in usable numbers, the callback question not dominated by one answer, enough escalations to score a rate against, enough non-counting callbacks to test Rule C-6 in both directions. No reader was involved: no model had been called and the free floors did not exist yet. The criterion is written into the generator beside the constant. ⚠︎ THE POLICY IS ILLUSTRATIVE. The nine rules, the corridor thresholds, the cutoffs and the two-attempt escalation bar reproduce no bank's payment policy, no scheme rule, no clearing deadline and no regulator's guidance, and they name none.

Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated payment-queue snapshot for an AI
use-case kit; it reproduces no bank's payment policy, no scheme rule and no real
customer, beneficiary, account or institution. The callback policy is ILLUSTRATIVE.

Instruction
----------------------------------------------------------------
Wire reference                 : WIR-0001
Ordering customer              : Brackenridge Foods SA (debit account ****7280)
Beneficiary                    : Torrent Packaging Pty
Beneficiary account            : ****7831
Beneficiary bank               : Pacific Rim Commercial
Beneficiary history            : 41 prior payments to this beneficiary from this account since 2024
Amount                         : 253,078.63 USD  (USD equivalent 253078.63)
Product / corridor             : USD-XB (USD cross-border)
Instruction channel            : SWIFT MT101 from the client's treasury system  [AUTHENTICATED on file]
Instruction received           : 2026-03-05 05:30
Value date                     : 2026-03-05
Cutoff for value today         : 2026-03-05 15:00

Callback Policy
----------------------------------------------------------------
Policy reference               : CBK-POL-2026-02 (illustrative)
Callback threshold on file     : 100,000.00 USD for USD-XB
Escalation after               : 2 counting attempts with no confirmation

Rule C-1  A callback is REQUIRED where the beneficiary's account number, IBAN, bank identifier or
          bank name has been CHANGED at any point after the instruction was first received, whatever
          the amount and whoever asked for the change.

The outcomeWhat a good result looks like

Every wire in the queue carries a disposition, a callback answer, a running count of callbacks that count, and a one-time escalation to the second approver -- re-derived every 90 minutes, with what changed since the last sweep carried forward in code rather than remembered by a person.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• You already know which wires are over the callback threshold and only need the list — a SQL predicate, or the b000 floor in this repo
  An amount against a threshold is a comparison. On this corpus b000 gets 63.89 pct of the callback answers right for nothing at all.
• Your case file is STRUCTURED -- change reasons come from a closed list and the callback system records which number was dialled and where it came from — the b002 floor: no keywords needed, code-only over your own fields
  The model's entire margin on this corpus is reading free prose. Take the prose away and there is nothing left to buy: b002 already gets 97.22 pct of the dispositions and 98.89 pct of the callback answers.
• Your change log and callback log are prose a payments clerk types, and a callback entry does not carry a machine-readable flag for whose number was dialled — this kit, with the carried state
  That is exactly the corpus these figures were measured on, and it is where the whole margin is: 707,767.70 USD released by the free floor against 0.00 USD by the model, and 8 of 13 escalations missed against 0.

And where nothing here is good enough:

• You want the kit to decide the release rather than brief a person, or to run unattended overnight — neither this kit nor any version of it
  It never releases, approves or cancels a payment and there is no setting that makes it. The one error in 180 readings is a disposition, and the honest reading of a 99.44 pct disposition figure on a synthetic corpus is that it is a synthetic corpus.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-23. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-23 — r001-wire-release. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with NO key configured reproduces: the 180 snapshots and the answer key (regenerated byte-identically by python3 tools/build_corpus.py from the committed seed), all fifteen pre-spend checks, all three free floors, the whole four-schedule cadence experiment, every committed result file, and the UI at 127.0.0.1:9011 including the button that replays what run r001 answered. What it cannot reproduce without a key is a new live reading -- and the read button says so in a sentence rather than failing.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per reading: one HTTP request carrying the assembled prompt, and the reply parsed to a JSON object. Measured over the 180 readings of run r001-wire-release at 10 concurrent wire chains, so it is a per-reading latency under load rather than a serial best case. The whole run took 212.1 seconds of wall clock. The free floors are not in this figure and would be zero: they make no request.

Current processWhat it replaces

Somebody working down the release queue at each sweep: opening each unreleased wire, reading the change log to decide whether anything in it touched the four fields the policy names, reading the callback log to decide which of those calls actually counted, adding them to however many counted at the last look, and remembering whether this one has already gone to the second approver.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• 3 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one wires (6 sweeps each) is

No split, and no chunking. The unit is a WIRE -- six consecutive sweeps processed strictly in order, each carrying the previous one's state -- so a train/test split over documents would break the chains that are the whole point. Every reading is scored; nothing is held out, because nothing is trained.

SetupWhat the setup figure measured

There is no index and no retrieval step -- the whole unreleased queue is re-read on every sweep. tools/build_corpus.py writes the 180 snapshots, the answer key and the event model in well under a second, once, and the result is committed.

LicenceLicence

MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved; every byte was written by the generator committed beside it.

Bring your ownBring your own release-queue snapshots

Point tools/build_corpus.py at your own release queue, or replace data/corpus/*.txt and data/gold.jsonl directly. The parser requires the eight section headings in src/segment.py::SECTIONS and evals/check_labels.py refuses to let a run spend until all eight parse in every document. Replace src/release.py's POLICY_TEXT and precedence with your own policy first: the shipped one is invented. data/wires.json is the event model behind the pages and is read only by evals/cadence.py -- nothing in src/ touches it and nothing in a prompt comes from it, so a corpus supplied directly simply loses the cadence experiment rather than breaking.

⚠︎ And what stops being true when you do: The measured figures on this page do not travel with your corpus, and three of them are corpus properties rather than model properties: the 90-minute cadence, a queue whose membership does not change between sweeps, and the collision between the change and callback phrasings, which is as fair as one author could make it. Re-run the three free floors on your own data before you read the model's margin as yours -- they cost nothing and they are the only honest baseline.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 8,799 characters assembled, 3,432 are instructions — 39% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

TWO MESSAGES, JOINED IN SEND ORDER. The provider receives a system message and a user message; prompt_verbatim prints both, separated by a blank line, because publishing only the second would be publishing most of a prompt. Replayed from the kit's own src/prompt.build() for WIR-0030-S1 with the carried state that reading was actually given, rather than logged by the run. What makes the replay checkable is that the section list it produces is the list the run recorded in sections_used -- Synthetic Record, Instruction, Callback Policy, Release Position, Instruction Change Log, Callback Attempts, Operational Notes -- and that the Customer Contact Record is absent from both. The token figures above are the run's AVERAGES over 180 readings, not this one reading's, and every snapshot in this corpus is within 4 pct of every other in size.

Full promptThe prompt in full

You are a scheduled outbound-wire release sweep in a bank's payment operations team. You apply a written callback policy to one wire at one sweep. You answer with one JSON object and no other text.

You are the scheduled release sweep on one bank's outbound wire queue. It wakes every 90 minutes,
re-reads EVERY wire still unreleased, and reports each one. You are reading ONE wire at ONE sweep.

The callback policy, the threshold on file and the cutoff are reproduced in the snapshot below.
Apply them exactly as written. Two things survive between sweeps and you cannot see the earlier
sweeps: whether a callback has already been established as required, and how many callback attempts
have already been made. What is known about them is stated under "Carried state" and is the only
history available to you. Do not assume anything about earlier sweeps beyond it.

How to decide:
 - "callback_required" is YES if ANY rule you can evaluate fires: a Rule C-1 change to the
   beneficiary's ACCOUNT NUMBER, IBAN, BANK IDENTIFIER or BANK NAME (nothing else on the instruction
   counts -- not the beneficiary's own name, not its address, not the charges option, not the value
   date, not our own reference); or the amount at or above the threshold on file; or a first payment
   to this beneficiary; or an unauthenticated channel. It is YES if the carried state says an
   earlier sweep already established it -- that requirement never lapses. It is UNDETERMINED only
   when no rule you can evaluate has fired AND a determinant is missing: no threshold on file for
   the corridor, or no channel recorded. Otherwise NO. Never assume a default threshold and never
   infer a channel (Rule C-5).
 - "callback_attempts" is the RUNNING TOTAL of attempts that COUNT under Rule C-6: the number the
   carried state gives you, PLUS the counting attempts in this window's log. An attempt counts only
   if it was placed to a number the bank already held on the customer record. An attempt placed to a
   number the instruction, a covering email or the sender supplied does NOT count, however the entry
   is worded and however confident the confirmation sounds. Count nothing twice: the log shows only
   the entries raised in THIS window.
 - A confirmation from a NON-counting attempt does not release the wire. A denial or a failure to
   confirm on a COUNTING attempt puts the wire into review.

Answer with a single JSON object and nothing else:

{"disposition": "RELEASE_OK|HOLD_CALLBACK|HOLD_REVIEW|MISSED_CUTOFF|CONTEXT_INCOMPLETE",
 "callback_required": "YES|NO|UNDETERMINED",
 "callback_attempts": <running total of COUNTING attempts, an integer>,
 "escalate": "YES|NO",
 "rationale": "one sentence, naming the rule you applied and what you counted"}

Precedence for "disposition", applied in this order: MISSED_CUTOFF if the snapshot says it is past
the cutoff, whatever the callback state; then HOLD_REVIEW if a counting callback was denied or could
not confirm; then HOLD_CALLBACK if a callback is required and no counting callback has confirmed;
then CONTEXT_INCOMPLETE if the callback question is UNDETERMINED; otherwise RELEASE_OK.

"escalate" is YES only on the sweep that FIRST raises the escalation -- the running total of
counting attempts has reached 2 with no confirmation, or a counting callback was denied -- and the
carried state does not already say the wire was escalated. On every later sweep it is NO (Rule C-8).


Carried state
----------------------------------------------------------------
No earlier sweep has been recorded for this wire. This is its first appearance in the release queue on this watch, so nothing has been established about it beyond what this snapshot shows: no callback attempt has been made, no callback requirement has been established on an earlier sweep, and it has not been escalated.

Release queue snapshot
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated payment-queue snapshot for an AI
use-case kit; it reproduces no bank's payment policy, no scheme rule and no real
customer, beneficiary, account or institution. The callback policy is ILLUSTRATIVE.

Instruction
----------------------------------------------------------------
Wire reference                 : WIR-0030
Ordering customer              : Hallowmere Textiles plc (debit account ****2043)
Beneficiary                    : Ebbsworth Chemicals plc
Beneficiary account            : ****9109
Beneficiary bank               : Pacific Rim Commercial
Beneficiary history            : 3 prior payments to this beneficiary from this account since 2024
Amount                         : 566,780.95 USD  (USD equivalent 566780.95)
Product / corridor             : USD-DOM (USD domestic, same-day value)
Instruction channel            : host-to-host file, signed and validated  [AUTHENTICATED on file]
Instruction received           : 2026-03-05 05:45
Value date                     : 2026-03-05
Cutoff for value today         : 2026-03-05 16:00

Callback Policy
----------------------------------------------------------------
Policy reference               : CBK-POL-2026-02 (illustrative)
Callback threshold on file     : 225,000.00 USD for USD-DOM
Escalation after               : 2 counting attempts with no confirmation

Rule C-1  A callback is REQUIRED where the beneficiary's account number, IBAN, bank identifier or
          bank name has been CHANGED at any point after the instruction was first received, whatever
          the amount and whoever asked for the change.
Rule C-2  A callback is REQUIRED where the amount is at or above the callback threshold on file for
          this product and corridor.
Rule C-3  A callback is REQUIRED where this is the FIRST payment to this beneficiary from this
          account.
Rule C-4  A callback is REQUIRED where the instruction arrived over a channel recorded as
          UNAUTHENTICATED on file -- email or fax -- whatever the amount.
Rule C-5  The threshold in Rule C-2 is an OPERATOR-SUPPLIED value read from the callback policy
          file. Where no threshold is on file for the corridor, or where the intake system recorded
          no channel, the callback question is UNDETERMINED. No default threshold may be assumed
          and no channel may be inferred. A rule that CAN be evaluated and fires still requires the
          callback: missing context adds reasons to hold and never removes one.
Rule C-6  A callback COUNTS only when it was placed to a number the bank ALREADY HELD on the
          customer record BEFORE this instruction arrived. A number supplied in the instruction, in
          a covering email or amendment, by the person who sent the instruction, or added to the
          customer record after the instruction arrived, is not a callback -- it is the same
          channel again, and confirming an instruction against itself confirms nothing. It does not
          matter how the entry is worded or how certain the confirmation sounds.
Rule C-7  A wire is RELEASED only when no callback is required, or a counting callback has
          CONFIRMED the beneficiary details. A counting callback that was DENIED, or where the
          contact could not confirm the details, puts the wire into review; it is not another
          unanswered attempt.
Rule C-8  After two counting attempts with no confirmation, or immediately on a denial, the wire is
          ESCALATED to the second approver. Escalation is raised ONCE per wire; a later sweep
          reports it already raised and does not raise a second one.
Rule C-9  A wire at or past its value-date cutoff has MISSED THE CUTOFF and cannot settle today,
          whatever its callback state. The callback question is still answered, because the wire
          will be re-dated and released tomorrow and the control still has to be satisfied.

Release Position
----------------------------------------------------------------
Sweep taken                    : 2026-03-05 08:00   (sweep 1 of this wire)
Sweep cadence                  : every 90 minutes (08:00 / 09:30 / 11:00 / 12:30 / 14:00 / 15:30 local)
Previous sweep                 : -- this is the first sweep of this wire
Minutes to cutoff at this sweep: 480
Past cutoff                    : no

Instruction Change Log
----------------------------------------------------------------
Entries raised in this window only.

07:15  beneficiary bank renamed on the instruction following its merger, new institution named

Callback Attempts
----------------------------------------------------------------
Entries raised in this window only.

08:00  used the onboarding contact, whose direct dial the payee's own email replaced last week; she confirmed the account

Operational Notes
----------------------------------------------------------------
Relationship manager flagged this payment for a follow-up on the charges option.

Raw responseThe raw response

{
 "disposition": "HOLD_CALLBACK",
 "callback_required": "YES",
 "callback_attempts": 0,
 "escalate": "NO",
 "rationale": "Rule C-2 (and Rule C-1) makes the callback required because the USD-DOM amount exceeds 225,000 and the beneficiary bank name changed, and the 08:00 call is not a counting attempt under Rule C-6 because the number used came from the payee's email, not a number the bank already held."
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Decide whether a bank wire may go out — 180 release-queue snapshots. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Every cell is compared for exact equality against a gold set computed by tools/build_corpus.py from the planted event model -- never from the rendered page. No model grades anything in this kit and no judge prompt exists. The running attempt count is compared as an integer; the other three fields are words from closed lists. A reply that did not parse counts as a MISS in every field, never as an exclusion, and 180 of 180 parsed on the scored run.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran all 373 calls is kept out of these tables per this estate's naming rule, so nothing here is what was paid. The real spend for this kit's whole build is on the shared call ledger, not on this page.

Grading unitWhat the grading figure prices

The GRADER costs nothing: evals/scoring.py makes no model call, and neither do the three free floors it is compared against, nor the four-schedule cadence experiment. What cost money is the two model ARMS -- 180 readings each -- and the 12-call calibration probe.

The gradersThree ways to grade

⚑ THE STRONGEST FREE FLOOR IS 97.22 PCT ON THE DISPOSITION AND 98.89 PCT ON THE CALLBACK QUESTION, FOR NOTHING, AND THE MODEL'S MARGIN IS NARROW AND EXPENSIVE. Three floors, all $0.00, all scored through the identical scorer, and they are built to separate two different things: b000 amountonly the threshold rule a payments hub already runs -- amount against a threshold, no memory, neither log read, and a 100,000 USD DEFAULT where the corridor has no row. 62.22 pct of dispositions, 24 false releases worth 1,195,456.03 USD, 0.0 pct context-incomplete recall. b001 amountonly-mem the same rule given the carried state. 77.22 pct of dispositions and 7 false releases worth 359,747.57 USD. THE GAP TO b000 IS WHAT MEMORY ALONE BUYS, WITH THE MODEL HELD OUT: +15.00 points of disposition accuracy, false releases 24 -> 7, and 835,708.46 USD of exposure removed. That is the largest single movement on this page and there is no model in it. b002 keyword-mem carried state, both logs parsed, Rule C-1 by keyword table and Rule C-6 by a second one, and an abstention where the corridor has no threshold. 97.22 pct of dispositions, 98.89 pct of callback answers, 92.78 pct of attempt counts. WHERE b002 LOSES IS RULE C-6, AND THAT IS WHERE THE MONEY IS. It counts 4 non-counting callbacks and releases 707,767.70 USD the key says must be held, and it misses 8 of 13 escalations, because a counting attempt it cannot recognise never advances the running total to two. Run r001 releases 0.00 USD and misses 0. ⚠︎ ONE ASYMMETRY FLATTERS THE FLOOR. b002 and the answer key share an implementation of the precedence -- both call src/release.step -- so the floor cannot misread the rule, it IS the rule, and only its inputs are guessed. The model can only read the prose. ⚠︎ AND THE FIRST VERSION OF b002 SCORED 99.44 PCT ON THE CALLBACK QUESTION BEFORE A SINGLE CALL WAS MADE, because its keyword tables enumerated the generator's own phrasings. That was a second answer key wearing a floor's clothes. Both the tables and the corpus were rewritten: the tables now list payments vocabulary and stop, and the phrasing pools collide in both directions.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same reading (one wire, at one sweep), the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

373 live calls were made for this kit and 373 returned something: 12 calibration (c000, at an 8,000-token ceiling on the two busiest wire chains), 180 scored, 180 the stateless control, and 1 for the live UI screenshot. NOTHING WAS DISCARDED, no run was re-fired and no result file was overwritten -- the one wrong cell on r001 is a rule ambiguity and is written up in the taxonomy rather than re-run away. The three free floors, the stub and the whole four-schedule cadence experiment cost $0.00 and made no request at all: provider none, 0 tokens, recorded as such in their own result files.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, AND MOST OF THEM ARE REASONING. 90.0 pct of this run's output (151,574 of 168,335) was provider-side reasoning left at the default, and output is 73.8 pct of the projected bill on the shared card. What you are paying for is the model weighing two prose judgements -- did this change touch one of the four fields Rule C-1 names, and did this callback count under Rule C-6.
• THE CADENCE, WHICH MULTIPLIES EVERYTHING ELSE. One reading is one call; the sweep wakes 6 times a day; the bill is rows x sweeps, not rows.
• THE UNRELEASED QUEUE, not the day's payment volume. A wire released at 08:00 leaves the watch and stops costing anything; a wire held all day is paid for 6 times.
• The snapshot itself, barely. Input averages 1989.1 tokens and every document in this corpus is within 4 pct of every other in size -- the callback policy block is reproduced in full on every page, which is most of it.

Your volumeWhat it costs at your volume

LINEAR IN ROWS x SWEEPS, AND THAT IS THE WHOLE WARNING. Ten times the unreleased queue is ten times the calls at the same cadence -- there is no batching, no cache and no early exit, because the queue is re-read whole on every sweep by design. A desk holding 300 wires on this 90-minute cadence is 1,800 calls a day, about 6.84 USD a day on the shared projection card. Nothing about the per-call price changes; the multiplier is the schedule, which is why this kit's environment ladder makes cadence a declared decision rather than a deployment detail -- and why the cheapest way to halve this bill is measured on this page rather than assumed.

Model choiceWhy this model

It is the tier the shared connection was already pointed at, and the kit's whole claim is that swapping is .env plus the same run again. ⚠︎ NOBODY HERE HAS RUN IT TWICE, so 'a second model would do better or worse' is not a statement this page can make -- it is in could_not_verify. What IS measured is that the tier chosen beats the strongest free floor on every band that costs money, and ties or loses to it on nothing.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every stateful number on these pages: 180 readings, one completion call each, one tier. The 180-call stateless control, the 12 calibration calls and the 1 live screenshot call are accounted for separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

12 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pythe corpus generator — a swap seam

Generates 30 outbound wires x 6 sweeps = 180 snapshots from a fixed seed (SEED = 20260923). Both logs are WINDOWED -- an entry is on the page of the sweep whose window it fell in and on no page after it -- and the change and callback phrasings are written to collide in both directions, so neither the word 'account' nor the phrase 'on file' decides anything. The gold labels are src/release.step's output over the planted events, never typed.

You change it to: The wires, the corridors, the cutoffs and every phrasing in both logs. Keep the eight section headings and supply your own gold.jsonl. The measured figures do not travel with it.

# Generate the corpus and the answer key. Deterministic: one seed, one corpus, forever.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
WIRES_JSON = os.path.join(HERE, "data", "wires.json")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
SEED = 20260923
WIRES = 30
DAY = "2026-03-05"
DAY_START = 6.0

src/release.pythe callback and release arithmetic — a swap seam

The policy as pure code: which triggers fired, how many attempts count, whether a counting callback confirmed or denied, whether the sweep is past the cutoff, and the precedence over the five dispositions. No model, no judgement. ⚠︎ The nine rules, the thresholds and the cutoffs in it are INVENTED and the kit ships that as a stated limitation -- see Data.why_this_corpus.

You change it to: POLICY_TEXT, ESCALATE_AFTER_ATTEMPTS and the precedence in step(). The policy is reproduced in full on all 180 pages precisely so it can be read, disbelieved and replaced with the one your own bank wrote.

# The callback-and-release rule as arithmetic. Pure code, no model, no library.

RELEASE_OK = "RELEASE_OK"
HOLD_CALLBACK = "HOLD_CALLBACK"
HOLD_REVIEW = "HOLD_REVIEW"
MISSED_CUTOFF = "MISSED_CUTOFF"
CONTEXT_INCOMPLETE = "CONTEXT_INCOMPLETE"
DISPOSITIONS = (RELEASE_OK, HOLD_CALLBACK, HOLD_REVIEW, MISSED_CUTOFF, CONTEXT_INCOMPLETE)
YES = "YES"
NO = "NO"
UNDETERMINED = "UNDETERMINED"

src/state.pythe carried state — a swap seam

SEAM 2 -- the thing that makes this a monitor. Six scalars per wire (counting attempts so far, whether a callback requirement has already been established, whether one has confirmed, whether one was denied, whether the wire has been escalated, and what the last sweep reported), written from the arithmetic and never from the model's reply, and rendered into one short paragraph for the prompt. It costs the same on sweep 60 as on sweep 2, which matters here because the watch wakes six times a day over the whole unreleased queue.

You change it to: What is carried between sweeps, and how it is worded to the model. Six scalars today. ⚠︎ THE WORDING IS UNMEASURED: the obvious experiment -- the same 180 readings with the state rendered as JSON instead of English -- costs one more 180-call run and has not been paid for. It is in could_not_verify.

# The carried state -- the thing that makes this a monitor and not another one-shot classifier.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEFAULT_PATH = os.path.join(HERE, "data", "state.json")
def load(path=DEFAULT_PATH):
def save(store, path=DEFAULT_PATH):
def for_wire(store, wire_id):
def describe(state):

src/select.pythe section filter

Decides which of the eight sections are sent. The Customer Contact Record -- two authorised signatories, their mandate direct dials and the callback passphrase -- is mapped by nothing and therefore never sent. Rule C-6 asks one question about those numbers and the ops log already answers it in words, so the numbers themselves buy nothing and risk everything.

# Pick which sections of a snapshot are sent. Pure code -- the last deterministic step before the

BANNER = "Synthetic Record"
INSTRUCTION = "Instruction"
POLICY = "Callback Policy"
POSITION = "Release Position"
CHANGES = "Instruction Change Log"
ATTEMPTS = "Callback Attempts"
CONTACT = "Customer Contact Record"
NOTES = "Operational Notes"
NEVER_SENT = (CONTACT,)

src/prompt.pythe prompt

Three parts: the fixed instruction, the carried-state paragraph, and the sections that got through the filter. The --stateless control replaces the middle block with one line and changes nothing else; evals/check_labels.py asserts the two builders differ on exactly one line.

# Assemble the one prompt this kit sends. No template engine, no framework -- string concatenation

INSTRUCTION = """\
def build(text, carried, stateless=False):

src/sweep.pyone sweep of one wire

One reading, one completion call. Parses the amount, the threshold, the channel flag, the beneficiary history and the minutes to cutoff off the page in code -- the model is never asked to read a number -- and records tokens, finish_reason and the provider's reasoning-token breakdown alongside the parsed answer.

# One wire, one scheduled sweep, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = ("You are a scheduled outbound-wire release sweep in a bank's payment operations team. "
MAX_TOKENS = 24000
FIELDS = ("disposition", "callback_required", "callback_attempts", "escalate")
def documents():
def wires():
def load_doc(doc_id):
def position_of(text):

evals/baseline.pythe three free floors — a swap seam

amountonly, amountonly-mem and keyword-mem, scored through the identical scorer at $0.00. The first two guess a 100,000 USD threshold where the corridor has none, which is the defect this row's guardrail names, and what that costs is measured rather than asserted: 0.0 pct context-incomplete recall against the model's 100.0 pct.

You change it to: CHANGE_WORDS, NOT_A_CHANGE, FROM_INSTRUCTION and FROM_RECORD -- the keyword tables the model is measured against. Widen them and the floor gets stronger and the model's margin narrows. ⚠︎ THE FIRST VERSION OF THIS FILE ENUMERATED THE GENERATOR'S PHRASINGS AND SCORED 99.44 PCT ON THE CALLBACK QUESTION BEFORE A SINGLE CALL WAS MADE. That was a second answer key wearing a floor's clothes; the shipped tables list payments vocabulary and stop.

# THE FREE FLOORS. Three of them, none a strawman, all of them $0.00.

MODES = ("amountonly", "amountonly-mem", "keyword-mem")
ASSUMED_THRESHOLD_USD = 100000.0
CHANGE_WORDS = ("account", "iban", "bic", "swift", "sort code", "routing", "bank")
NOT_A_CHANGE = ("no change", "unchanged", "verified")
FROM_INSTRUCTION = ("covering email", "on the instruction", "the sender", "sender's",
FROM_RECORD = ("on file", "on the record", "customer record", "mandate", "account opening",
CONFIRM_WORDS = ("confirmed", "read back", "passphrase matched")
DENY_WORDS = ("did not send", "denies", "could not confirm", "does not recognise",
def _num(pat, text, cast=float):

evals/cadence.pythe cadence experiment

Re-derives the answer key on four schedules and two query implementations, and costs $0.00 -- no provider is called for any of it. It is what produced this kit's largest finding: the same skipped sweep costs a value date under one implementation and 207,852.41 USD of unchecked payments under the other.

# What a missed sweep costs. Free -- no calls, no key, no model anywhere near it.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
OUT = os.path.join(RESULTS, "eval-cadence-wire-release.json")
BC = importlib.util.module_from_spec(_spec)
FULL = BC.SWEEP_HOURS
SCHEDULES = [
def marks(w, schedule, fixed_window=False):
def _escalations_total(wires):
def _median(xs):

evals/scoring.pythe scorer

Exact match per cell against the answer key, plus the two failure directions counted APART -- false releases over the 80 readings that must be held, unnecessary holds over the 43 that may go -- and the dollars beside each. No model grades anything.

# Score a run against the answer key. Pure code, exact match per cell. No model grades anything.

FIELDS = ("disposition", "callback_required", "callback_attempts", "escalate")
HOLDS = ("HOLD_CALLBACK", "HOLD_REVIEW")
def _pct(n, d):
def _int(v):
def score(records, golds):

evals/check_labels.pythe pre-spend checks

Fifteen claims this kit's page makes, checked against the corpus that will actually be sent, in seconds and for nothing. Two of them are the privacy guard red-proven in both directions; one asserts no passphrase, direct dial or signatory name appears in any of the 180 assembled prompts; one asserts every log entry appears on exactly ONE sweep's page, which is what makes every memory claim on this page falsifiable.

# Everything that must be true BEFORE a run is allowed to spend. Free, and it runs in seconds.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
WIRES_JSON = os.path.join(HERE, "data", "wires.json")
FAILED = []
def check(name, ok, detail=""):
def main():

src/app.pythe local UI

http.server, no dependency, port 9011. Renders with no key: the snapshot, the carried state and the free floor are computed locally, and a second button replays what run r001 actually answered off the committed result file, labelled as a replay.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
PORT = int(os.environ.get("PORT", "9011"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-wire-release")
GOLD_ROWS = {r["doc_id"]: r for r in
def carried_for(doc_id):
class H(BaseHTTPRequestHandler):
def main():

src/adapters/__init__.pythe model adapter — a swap seam

SEAM 1. Raw HTTP for every provider, retry on 429/5xx and on transport failures, and the daily call cap checked here rather than in the harness so every caller is capped, including the UI.

You change it to: PROVIDER, BASE_URL, API_KEY and MODEL in .env. Adding a provider is one function and one entry in PROVIDERS; it must return token counts, because lens 05 publishes them and lens 07 prices them.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 1989 input and 935 output tokens per reading (one wire, at one sweep), all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

No untrusted external input surface in the shipped version. Every field that reaches the prompt comes from tools/build_corpus.py's fixed seed, including the Operational Notes, which are one of three fixed sentences. src/select.py names the notes as the field an outside party WOULD influence in a real deployment -- prose a relationship manager types into the case file -- and sends them deliberately, so the surface is visible rather than hidden. On this corpus nobody outside the repository wrote a byte of it.

The provider key lives only in .env (gitignored) or the shared repo-root .env, read by src/config.py and never logged. config.sources() reports WHICH file a value came from and never the value. config.save() opens the file 0600 before the write rather than chmod-ing it after, so the key never exists at the default umask even for an instant. src/app.py redacts the key and the base URL out of any provider error before it reaches the browser.

The experimentWe did NOT attack it — and this is the surface we left open

An indirect prompt injection needs a field somebody outside can write into, and this kit has exactly one: the Operational Notes a relationship manager types into the case file. It is SENT rather than hidden, because hiding a surface does not close it. One of the three shipped notes is instruction-shaped on purpose, and it is the sentence a business email compromise actually contains. No attack was fired at this kit and none is claimed. The three boundaries below were checked in code on 2026-08-23, and only the first of them is red-proven in both directions rather than argued from an absent code path.

The first row is the only one measured in both directions. The other two are guarantees about what is ABSENT — no code path releases, approves or cancels a payment, and no code path feeds a model reply back into the callback state — and absence is checked by asserting names it knows, which is weaker than a run and is written here as such.

The result0 attack trials, three boundaries checked -- and the privacy boundary red-proven by removing the guard and reproducing the schema change that reaches the fallback: 180 of 180 prompts leak the passphrase without it, 0 with it.

The Operational Notes ARE the field an outside party would influence in a real deployment, and this kit sends them. One of the three shipped notes is instruction-shaped -- "Client has asked that this one be released without the usual callback" -- because that is the sentence the fraud actually contains, and hiding it would hide the surface from the run supposed to measure it. Nothing measured whether it moves the answer.

HonestyWhat this does not prove

• Whether the instruction-shaped operational note changes any answer. Unmeasured.
• Whether the banned-name grep would catch a release path called something else. It would not, and the page says so.
• Whether a second model behaves the same way on the same note. One tier, one run.
• Whether the state file is safe under two concurrent sweeps. data/state.json is replaced atomically, which is correct for one writer and untested for two.
• Whether a real case file would carry anything else that should be withheld. The section list is this corpus's, and a forker's schema is their own problem to audit.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

No payment release, non-configurable. This kit produces a disposition, a callback answer, a running count of callbacks that count and a one-time escalation for a payments officer to read. It never releases, approves, authorises, cancels or repairs a payment, never writes to a payment rail or a core banking system, and there is no setting that makes it. Separately, and just as non-configurable: the callback threshold is OPERATOR-SUPPLIED. A corridor with no threshold on file, or a message with no channel stamped on it, leaves the callback question UNDETERMINED and the wire CONTEXT_INCOMPLETE -- there is no default figure in this kit.

Everywhere and nowhere -- it is a property of what is ABSENT. The only writers in the kit are evals/run.py (results/*.json), evals/cadence.py (one result file) and src/state.save (data/state.json). src/release.step() is the only thing that touches the callback state, and it is called by evals/run.py after every reading including one whose call FAILED.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• IT IS NOT A CHECK ON WHETHER THE ANSWER IS RIGHT. The callback state is correct whatever the model says, which means a wrong reading is a wrong watchlist row and not a corrupted history. Those are different problems and only the first one is on this page.
• IT IS NOT A SCHEDULER. evals/run.py is invoked; it is not woken. Nothing here detects a missed sweep, back-fills it or marks its readings late -- which is the half of a monitor this variant exists to name, and on this kit the hole has a price on it: see the cadence experiment.
• IT IS NOT AN AUTHORITY ON PAYMENT CONTROLS. The nine rules, the thresholds, the cutoffs and the two-attempt escalation bar are invented for this kit and reproduce no bank's policy, no scheme rule, no clearing deadline and no regulator's guidance.
• IT IS NOT A SANCTIONS, AML OR FRAUD SCREEN. It reads a callback control and nothing else. A wire can pass every rule on this page and still be a payment nobody should make.

WatchedWhat is watched, and why that one

The kit measures 36 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

22 measured by the latest run 14 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (cadence_minutes 90, context_incomplete_cells 27, documents 180, escalate_cells 13, hold_cells 80, memory_cells 86, pre_cutoff_cells 150, quiet_cells 167, readings_scored 180, release_cells 43, stateless False, value_at_risk_usd_gold 39719857.7, wires 30) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

7 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

triage · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 3 runs measured the same system — they differ on floor, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

triage · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on context_incomplete_cells, documents, escalate_cells, hold_cells, max_tokens, memory_cells, pre_cutoff_cells, quiet_cells, readings_scored, release_cells, stateless, value_at_risk_usd_gold, wires — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 22 chips that all say so.

DeviationsWhat deviated

0 breaches across 7 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-run evals/check_labels.py (free, under a second) on any change to tools/build_corpus.py, src/release.py, src/segment.py or src/select.py -- it refuses to let a run spend if the corpus, the answer key, the experimental control, the privacy guard or the no-release property has moved. Re-run evals/cadence.py on any change to the schedule or the window rule; it is free and takes under a second. Re-run the scored eval AND its stateless control together (paid, 360 calls) on any change to src/prompt.py or src/state.py: the headline is a difference, so one arm re-run alone is not comparable with the other's old figure. All three free floors cost nothing and should be re-run on every change, in both directions -- a floor that suddenly improves is a corpus that got easier.

In one lineNo framework, deliberately

No framework and no dependencies at all -- json, os, re, random, argparse, urllib, http.server, importlib and concurrent.futures from the standard library. requirements.txt names nothing and says in a comment that the emptiness is load-bearing: a line there means something under src/ or evals/ imports it.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph, and the one place it looks like one is a straight line. Each wire is a chain of 6 readings -- 08:00, 09:30, 11:00, 12:30, 14:00, 15:30 -- with no branching and exactly one edge between consecutive sweeps, carrying six scalars. Different wires never touch. A framework would add an orchestrator to a for-loop that already runs 30 chains wide.

The other sideWhat a framework costs you

• No scheduler. This is a watch that carries state and is still INVOKED, not woken -- and on this kit the cadence is not a deployment detail, it is a line item: see the cadence experiment.
• No missed-sweep detection, no back-fill, no late marking. All three are what a scheduling engine gives you and none of them exists here. What they are WORTH is measured, for nothing.
• No concurrency model for the state store. data/state.json is one file replaced atomically, which is right for one process and is not a design for a scheduler.
• No retry policy above the HTTP layer. src/adapters retries a 429 or a 5xx four times with backoff; a reading that still fails is recorded as a failure and the chain advances, which is correct for an eval and is not an operational answer.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-wire-release on the same tier, memory removed (THE CONTROL), 2026-08-23. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 5 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

3 runs not plotted. b000-wire-release-amountonly, b001-wire-release-amountonlymem, b002-wire-release-keywordmem recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the population is re-read whole on each scheduled run, and the prior run's carried state is what makes a reading a CHANGE.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved; every byte was written by the generator committed beside it. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe disposition, the callback answer, the counting-attempt total and the escalate call, per reading, exact match against the computed answer key

For each of the 180 readings and each of the four answered fields, did the reply equal the computed answer key? Three of the four are words from closed lists; the attempt count is compared as an integer, and 0 is a meaningful answer here -- it is what every wire with no counting callback gets -- so a parse failure must not be scored as one.

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model. The same function all three free floors are scored through.

Every grader on these pages scored the same 180 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 180 per field. A reading whose reply did not parse counts as a MISS in every field, never as an exclusion.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, computed by tools/build_corpus.py over the planted event model at generation time and re-derived from src/release.step by evals/check_labels.py before any run may spend. This grader IS the reference, so its own TPR and TNR are not measured -- scoring it against itself would be circular.

How tight can the band be? No tuned threshold anywhere on the model path -- the reply decides. The denominators are printed beside every rate because two of them are small: 13 readings must escalate and 27 have nothing on file.

Cadence: Re-run evals/check_labels.py (free, under a second) on any change to tools/build_corpus.py, src/release.py, src/segment.py or src/select.py -- it refuses to let a run spend if the key, the control or the privacy guard has moved. Re-run the scored eval AND its stateless control together (paid, 360 calls) on any change to src/prompt.py or src/state.py: the headline is a difference, so one arm re-run alone is not comparable with the other's old figure. All three floors are free and should be re-run on any change at all -- a floor that suddenly improves is a corpus that got easier.

The decisionWhen to reach for it

In one lineThe two failure directions, counted apart, with their dollars

A FALSE RELEASE is a reading whose key says hold and whose answer says RELEASE_OK: the money leaves with its control unsatisfied and a recall is a request, not a right. An UNNECESSARY HOLD is the mirror: the key says the wire may go and the reading holds it, so it misses its value date. Both are counted, and the FACE VALUE of the wires in each is summed beside the count.

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model.

Every grader on these pages scored the same 180 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

false_release_pct = false releases / 80 readings that must be held. unnecessary_hold_pct = unnecessary holds / 43 readings that may go. Never averaged, never combined into an F-score.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the same data/gold.jsonl. A reading is in the FALSE RELEASE population if its key is HOLD_CALLBACK or HOLD_REVIEW, and in the UNNECESSARY HOLD population if its key is RELEASE_OK. The two populations are disjoint and neither includes the 30 readings past their cutoff or the 27 with nothing on file.

How tight can the band be? No threshold. A reading is in one population or the other by its KEY, and the reply either matches or does not.

Cadence: Free and re-run with the cell scorer, in the same pass. The DOLLAR figures move with the corpus as well as with the reader, so they are only comparable between arms of the same dataset_version -- re-generate the corpus and every USD figure on this page is a different question.

The decisionWhen to reach for it

In one lineContext-incomplete recall -- the guardrail, scored rather than asserted

Of the 27 readings whose corridor carries no callback threshold on file or whose message was never stamped with a channel, how many were reported CONTEXT_INCOMPLETE with the callback question UNDETERMINED -- rather than answered against a guessed number?

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model.

Every grader on these pages scored the same 180 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

hits / 27.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the 27 readings whose corridor carries no threshold on file or whose message was never stamped with a channel, taken from the generator's own flaw flag rather than from the rendered page.

How tight can the band be? No threshold. The reading either reported CONTEXT_INCOMPLETE with the callback question UNDETERMINED or it did not.

Cadence: Free and re-run with the cell scorer. ⚠︎ Re-run it FIRST after any change to tools/build_corpus.py: the denominator is a corpus property, and a generator change that quietly removes the flawed corridors would take this rate to 100 pct over zero cells, which reads as a pass and is a switched-off guardrail.

The decisionWhen to reach for it