The business caseThe problem this solves

"Which trades are still failing" is a SELECT and nobody needs a language model for it. The question a settlements desk actually has to answer every evening is the next one, and it has a hard, one-way clock on it: how many SETTLEMENT DAYS has this fail aged AT ITS OWN PLACE OF SETTLEMENT -- not calendar days, and not the desk's own business days -- and is today the day the mandatory buy-in falls due. Three things make that unanswerable from the row in front of you. The market where it settles may have been shut on a day the desk was open. The line may have been un-settleable for reasons that are nobody's counterparty's fault, which do not age the fail, mixed in with days the delivering party simply did not deliver, which do. And the exceptions feed keeps one day, so an event that opened before this window is not on this page at all. Somebody working the evening fails list by hand: opening each open fail, finding its place of settlement, pulling that market's calendar, counting settlement days from the intended settlement date, deciding for each exceptions line whether the market was shut or the counterparty was short, subtracting the first kind and not the second, projecting the remaining days forward across the next holiday, and remembering which fails already have a buy-in instructed. On this book that is 23 readings an evening, five evenings a week.

The inputThe actual fail snapshots

The corpus is 112 fail snapshots, 0.68 MB (json 1 · jsonl 2 · txt 201). A SETTLEMENT FAILS FILE IS BOTH THE MOST COMMERCIALLY SENSITIVE AND ONE OF THE MOST REGULATED BOOKS ON A DESK. It names the counterparty, the instrument, the size and the price of every trade a firm could not settle -- a list of who cannot deliver what is information that moves a price -- and there is no public one at any granularity that would let a monitor be tested. The alternative, a scrubbed real book, is worse: scrubbing is exactly the step that changes what the model is asked to read, and on this book the counterparty is half of what the line is about. So the corpus is invented and the generator ships beside it. See data/SOURCES.md.

Synthetic Record
------------------------------------------------------------------------
Every field below is invented. This is a generated settlement-fail snapshot for an AI
use-case kit. It reproduces no real trade, counterparty, fund, depository or market, and
the buy-in terms are ILLUSTRATIVE.

Fail
------------------------------------------------------------------------
Fail reference                         : FAIL-0001
Trade reference                        : TRD-898073
Counterparty reference                 : CPTY-3259
Instrument reference                   : SYN-1954  (ordinary shares)
Instrument class                       : Liquid equity
Place of settlement                    : NGX -- Northgate Exchange (Northgate Central Depository)
Direction                              : RECEIVE -- we are the receiving party; the
                                         delivering party has not delivered
Intended settlement date (ISD)         : 2026-02-27
Original quantity                      : 20,000
Residual failing quantity              : 20,000
Price on file                          : 167.74 USD
Residual exposure at the price on file : 3,354,800.00 USD

Buy-in Terms
------------------------------------------------------------------------
Settlement agreement                   : SSA-5801-2025
Extension period on file               : 4 settlement days from the intended settlement date
Buy-in agent                           : standing appointment; instructed by this desk

Rule S-1  A fail ages in SETTLEMENT DAYS at its PLACE OF SETTLEMENT, counted from the day AFTER the
          intended settlement date up to and including the snapshot day. A day on which that market
          does not settle -- a weekend, or a holiday in that market -- does not age the fail, even

The outcomeWhat a good result looks like

Every open fail carries a counted age in settlement days, an exact buy-in deadline date, and one clear instruct-or-hold call, and exactly one buy-in is instructed per fail on the run at which the extension period is first found exhausted. A fail whose agreement carries no extension period is reported CONTEXT_INCOMPLETE and ages nothing, rather than being aged against a default nobody agreed to.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• You already know which fails are past their extension period and only need the list — a SQL query against your settlements book, or the b000 floor in this repo
  Days elapsed against a threshold is arithmetic. On this corpus b000 gets 59.82 pct of statuses right for nothing at all -- it is only the DEADLINE DATE it cannot produce.
• You need a safety score for the whole kit rather than one abstention rule — a red-team run this kit does not have
  context_incomplete_recall_pct measures ONE stated abstention rule on 10 readings. It is a guardrail score, not a safety score.
• You want a number for what YOUR missed runs cost — evals/cadence.py, re-run on your own book -- it costs nothing and needs no key
  The crossings-lost figure is a fact about which fails in THIS book settle in full the morning after they cross. Yours will differ and the code is free.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-23. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-23 — r001-settle-fail. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with NO key configured reproduces: both corpora and both answer keys (regenerated from the seed by python3 tools/build_corpus.py, byte-identical), every pre-flight check in evals/check_labels.py, all four free floors, the free missed-run floor arm, the whole of evals/cadence.py including the crossings lost permanently, and the UI on both schedules with the free-floor column populated and the scored runs replayable off the committed result files. What it cannot reproduce without a key is a live model column.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per reading: one HTTP request carrying the assembled prompt, and the reply parsed to a status, an age, a deadline date and an instruct-or-hold call. It does NOT include the corpus build, the label checks or the scorer, none of which call anything. Measured over the 112 readings of r001-settle-fail with 16 concurrent fail chains, so it is the latency a single reading saw under that concurrency, not a serial figure.

Current processWhat it replaces

Somebody working the evening fails list by hand: opening each open fail, finding its place of settlement, pulling that market's calendar, counting settlement days from the intended settlement date, deciding for each exceptions line whether the market was shut or the counterparty was short, subtracting the first kind and not the second, projecting the remaining days forward across the next holiday, and remembering which fails already have a buy-in instructed. On this book that is 23 readings an evening, five evenings a week.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• 3 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one fails (up to 5 scheduled runs each) is

No split, and no chunking. The unit is a FAIL -- up to five consecutive scheduled runs processed strictly in order, because run 4's prompt contains a count produced by run 3. Different fails are independent and run concurrently. There is no train or dev set: nothing is fitted, and the four free floors were written from settlements vocabulary rather than tuned against this corpus.

SetupWhat the setup figure measured

There is no index and no retrieval step -- every open fail is re-read whole on each scheduled run. tools/build_corpus.py writes both corpora, both answer keys and the stats file in about a second, entirely offline.

LicenceLicence

MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved; every field was written by the generator committed beside it.

Bring your ownBring your own fail snapshots

Point tools/build_corpus.py at your own book, or replace data/corpus/*.txt and data/gold.jsonl directly. Keep the eight section headings (src/segment.SECTIONS -- the parser asserts all eight before a run may spend), keep the two dated blocks in the Settlement Calendar section in the printed form, and put your own markets in src/marketcal.MARKETS. The gold row shape is one line of data/gold.jsonl.

⚠︎ And what stops being true when you do: The measured figures on this page do not travel with your corpus, and two of them are corpus properties rather than model properties. The 95.54 pct the strongest free floor scores is a fact about thirteen exceptions phrasings and four holiday calendars. The 2 crossings lost to one missed run is a fact about which fails happen to settle in full the next morning. Re-run all seven arms on your book before quoting anything here.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 10,652 characters assembled, 4,207 are instructions — 39% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

TWO MESSAGES, JOINED IN SEND ORDER. The provider receives a system message and a user message; prompt_verbatim prints both, separated by a blank line, because publishing only the second would be publishing most of a prompt. Replayed from the kit's own src/prompt.build() for FAIL-0016-R3 with the carried state that reading was actually given, rather than logged by the run. What makes the replay checkable is that the section list it produces is the list the run recorded in sections_used -- Synthetic Record, Fail, Buy-in Terms, Ageing Position, Settlement Calendar, Activity Log, Operational Notes -- and that src/prompt.build() is the only thing in the kit that assembles a prompt, so there is no second path a run could have taken. Every one of the 10 parts below is a literal substring of the verbatim prompt, in ascending order, and the builder asserts it. ⚠︎ WHAT IT IS NOT CHECKED AGAINST, SAID PLAINLY: the run's own prompt_parts records ONE size, 10663 characters, belonging to whichever reading finished first under 16 concurrent workers -- not to FAIL-0016-R3, whose assembled user message is 10678 characters. The harness stores one exemplar, not one per reading, so no per-reading size comparison is available and none is claimed.

Full promptThe prompt in full

You are a scheduled settlement-fail watch. You apply a written buy-in rule to one open fail at one scheduled run. You answer with one JSON object and no other text.

You are the scheduled settlement-fail watch for one asset manager's settlements desk. It wakes once
every business day, after the settlement cycle has closed, re-reads EVERY fail still open on the
book, and reports each one. You are reading ONE fail at ONE scheduled run.

The buy-in rules, the extension period on file and the place-of-settlement calendar are reproduced
in the snapshot below. Apply them exactly as written. The fail's AGE is a RUNNING COUNT of
settlement days that survives between scheduled runs, and you cannot see the earlier runs; what is
known about them is stated under "Carried state" and is the only history available to you. Do not
assume anything about earlier runs beyond it.

How to count:
 - "Settlement days at <market> in this window" is how many days in this window that market
   settled. It is already computed for you from the calendar; do not re-derive it.
 - From those days, SUBTRACT every one covered by a SUSPENSIVE event (Rule S-3). Do NOT subtract a
   day lost to the delivering counterparty -- no stock, a blocked account, a mis-instruction,
   funding that did not arrive. Those days age the fail in full, because they are what the buy-in
   exists for. Deciding which kind an entry is means reading what it says, not which words it
   uses: an entry can be written entirely in counterparty language and still describe a market
   that was closed to everybody, and the other way round.
 - A suspensive event runs from its opening entry INCLUSIVE to its closing entry EXCLUSIVE. The
   Activity Log shows only the entries raised on THIS snapshot day. A closing entry with no opening
   entry belongs to an event that was already running: the carried state says whether one was open
   and therefore whether the earlier settlement days in this window were suspended.
 - ADD what is left to the age the carried state says was already counted.
 - The buy-in deadline is the settlement day on which that count reaches the extension period.
   Project it forward over the "Forward" calendar in the snapshot, counting only days marked
   SETTLES. If a suspensive event is open at this snapshot, skip the FIRST such day (Rule S-5).
 - Once the count has reached the extension period, the deadline is FIXED at the day it reached it
   and never moves again (Rule S-6). If the carried state already gives you a fixed deadline,
   repeat it. If the count reaches the extension period on THIS run, identify the day it reached
   it from the "Since the previous scheduled run" calendar in this snapshot -- when more than one
   settlement day fell in this window, that day may be earlier than the snapshot day.
 - A partial settlement does NOT re-base the intended settlement date (Rule S-8). The residual
   keeps the original ISD, the original age and the original deadline.
 - If no extension period is on file, or no intended settlement date is captured, the fail is
   CONTEXT_INCOMPLETE: status CONTEXT_INCOMPLETE, age 0, buyin_deadline NOT_DETERMINABLE,
   raise_buyin NO. Never assume a default extension period (Rule S-2).

Answer with a single JSON object and nothing else:

{"status": "AGEING|DUE_NEXT_RUN|BUYIN_DUE|SUSPENDED|CONTEXT_INCOMPLETE",
 "age_settlement_days": <whole number of settlement days now counted against this fail>,
 "buyin_deadline": "YYYY-MM-DD", or "NOT_DETERMINABLE",
 "raise_buyin": "YES|NO",
 "rationale": "one sentence, naming the rule you applied and the days you counted"}

Precedence for "status", applied in this order: CONTEXT_INCOMPLETE beats everything; then BUYIN_DUE
if the counted age has reached the extension period, even if a suspensive event is open right now;
then SUSPENDED if a suspensive event is open at this snapshot; then DUE_NEXT_RUN if the deadline
falls on or before the next scheduled run named in the snapshot; otherwise AGEING.

"raise_buyin" is YES only on the run that FIRST finds the extension period exhausted -- the count
has reached it and the carried state does not already give a fixed deadline. On every later run it
is NO (Rule S-7).

Carried state
------------------------------------------------------------------------
As at the previous scheduled run, 6 settlement day(s) had already been counted against this fail at its place of settlement, suspensive days already excluded. No suspensive event was open when that run ended. The extension period has not yet been found exhausted, so no deadline has been fixed and no buy-in has been instructed. It was reported DUE_NEXT_RUN.

Fail snapshot
------------------------------------------------------------------------
Synthetic Record
------------------------------------------------------------------------
Every field below is invented. This is a generated settlement-fail snapshot for an AI
use-case kit. It reproduces no real trade, counterparty, fund, depository or market, and
the buy-in terms are ILLUSTRATIVE.

Fail
------------------------------------------------------------------------
Fail reference                         : FAIL-0016
Trade reference                        : TRD-220266
Counterparty reference                 : CPTY-2967
Instrument reference                   : SYN-2285  (senior unsecured notes)
Instrument class                       : Corporate debt
Place of settlement                    : NGX -- Northgate Exchange (Northgate Central Depository)
Direction                              : RECEIVE -- we are the receiving party; the
                                         delivering party has not delivered
Intended settlement date (ISD)         : 2026-02-26
Original quantity                      : 5,000
Residual failing quantity              : 5,000
Price on file                          : 42.34 USD
Residual exposure at the price on file : 211,700.00 USD

Buy-in Terms
------------------------------------------------------------------------
Settlement agreement                   : SSA-2623-2025
Extension period on file               : 7 settlement days from the intended settlement date
Buy-in agent                           : standing appointment; instructed by this desk

Rule S-1  A fail ages in SETTLEMENT DAYS at its PLACE OF SETTLEMENT, counted from the day AFTER the
          intended settlement date up to and including the snapshot day. A day on which that market
          does not settle -- a weekend, or a holiday in that market -- does not age the fail, even
          if the watching desk is open that day.
Rule S-2  The extension period is an AGREEMENT-SUPPLIED value read from the settlement agreement.
          Where no extension period is on file, or where no intended settlement date is captured,
          the fail is CONTEXT_INCOMPLETE: it does not age, no deadline is determinable, no buy-in
          is instructed, and no default extension period may be assumed.
Rule S-3  A settlement day on which the line could not settle AT ALL -- a depository blackout, a
          market suspension, a settlement-engine outage, a corporate-action freeze -- is SUSPENSIVE
          and does not age the fail. A day lost to the delivering party -- no stock in the account,
          a blocked account, a mis-instruction, funding that did not arrive -- is NOT suspensive and
          ages the fail in full. That is what the buy-in exists for.
Rule S-4  A suspensive event runs from the day its opening entry is raised, INCLUSIVE, to the day
          its closing entry is raised, EXCLUSIVE. The activity log shows only the entries raised on
          the snapshot day, so an event that opened earlier is known only from the carried state.
Rule S-5  The buy-in deadline is the settlement day on which the counted age reaches the extension
          period, projected forward over the place-of-settlement calendar. A suspensive event open
          at the snapshot is projected to cost ONE further settlement day; if it runs longer, a
          later run moves the deadline out.
Rule S-6  Once the counted age has reached the extension period, the deadline is FIXED at the day
          it reached it and never moves again. A suspension after that day does not extend it.
Rule S-7  A buy-in is instructed ONCE per fail, on the first scheduled run at which the extension
          period is found exhausted. A later run reports the requirement already live; it does not
          instruct a second buy-in.
Rule S-8  A PARTIAL SETTLEMENT DOES NOT RE-BASE THE INTENDED SETTLEMENT DATE. The residual quantity
          keeps the original ISD, the original counted age and the original deadline.
Rule S-9  A fail is DUE_NEXT_RUN while its deadline falls on or before the next scheduled run --
          that is, when the requirement will become live before the desk looks again.
Precedence, applied in this order: CONTEXT_INCOMPLETE, then BUYIN_DUE, then SUSPENDED, then
          DUE_NEXT_RUN, then AGEING.

Ageing Position
------------------------------------------------------------------------
Snapshot taken                         : 2026-03-09 18:00   (scheduled run 3 of this watch)
Watch cadence                          : every business day, 18:00, at the watching desk
Previous scheduled run                 : 2026-03-06 18:00
Next scheduled run                     : 2026-03-10 18:00
Settlement days at NGX in this window  : 1
Settlement days since ISD, before any suspension : 7

Settlement Calendar
------------------------------------------------------------------------
Place of settlement                    : NGX -- Northgate Exchange
NGX settles Monday to Friday except the dates marked CLOSED below.

Since the previous scheduled run
  2026-03-07 Sat  CLOSED (weekend)
  2026-03-08 Sun  CLOSED (weekend)
  2026-03-09 Mon  SETTLES

Forward, from the next calendar day
  2026-03-10 Tue  SETTLES
  2026-03-11 Wed  SETTLES
  2026-03-12 Thu  SETTLES
  2026-03-13 Fri  SETTLES
  2026-03-14 Sat  CLOSED (weekend)
  2026-03-15 Sun  CLOSED (weekend)
  2026-03-16 Mon  SETTLES
  2026-03-17 Tue  SETTLES
  2026-03-18 Wed  SETTLES
  2026-03-19 Thu  SETTLES
  2026-03-20 Fri  CLOSED (market holiday: Spring Bank Holiday)
  2026-03-21 Sat  CLOSED (weekend)
  2026-03-22 Sun  CLOSED (weekend)
  2026-03-23 Mon  SETTLES
  2026-03-24 Tue  SETTLES
  2026-03-25 Wed  SETTLES

Activity Log
------------------------------------------------------------------------
Entries raised on this snapshot day only. This feed does not retain earlier days.

07:41  neither we nor the counterparty can put this through: the line has been shut to all instructions since the notice went out this morning

Operational Notes
------------------------------------------------------------------------
Nothing outstanding on the pre-matching check; shape and place of settlement both agree.

Raw responseThe raw response

{
 "status": "SUSPENDED",
 "age_settlement_days": 6,
 "buyin_deadline": "2026-03-11",
 "raise_buyin": "NO",
 "rationale": "Applied Rule S-3 to exclude 2026-03-09 as a suspensive shutdown and Rule S-5 to skip 2026-03-10, leaving age at 6 of 7 extension days with the projected deadline at 2026-03-11."
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch failing trades before the buy-in deadline — 112 fail snapshots. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Every cell is compared for exact equality against a gold set computed by tools/build_corpus.py from the fail model and the market calendars, never read back off the rendered page. Four fields per reading: the status, the counted age in settlement days, the buy-in deadline as an exact ISO date, and the instruct-or-hold call. A date is compared as a string, so being one day early is exactly as wrong as being a month early on the hit rate -- which is why the scorer ALSO counts the signed error in days and separates early from late.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran all 275 calls is kept out of these tables per this estate's naming rule, so nothing here is what was paid. The real spend for this kit's whole build is on the shared call ledger, not on this page.

Grading unitWhat the grading figure prices

The GRADER costs nothing: evals/scoring.py makes no model call, and neither do the four free floors, evals/cadence.py or evals/check_labels.py. No model judges anything anywhere in this kit. The only money is the readings themselves.

The gradersThree ways to grade

⚑ THE MODEL BEATS THE STRONGEST FREE FLOOR, AND THE MARGIN IS 4.46 POINTS ON THE DEADLINE. 100.0 pct against 95.54 pct on the exact buy-in deadline date, 100.0 against 97.32 on the status. That is the honest headline and it is a narrow one: a team that cannot spend should run marketcal-mem and be nearly all the way there. ⚑ THERE ARE FOUR FLOORS BECAUSE THERE ARE THREE SEPARATE THINGS TO PRICE. b000 caldays -- the aged-fails report a desk already runs. 8.93 pct on the deadline. It is not a strawman; it is what a threshold report configured once and forgotten actually does, including guessing a default extension period where the agreement carries none, which is the guardrail Rule S-2 forbids: 0.0 pct context-incomplete recall. b001 caldays-mem -- the same rule with memory. Identical on status, age and deadline, to the decimal. THE ONLY THING MEMORY BUYS A CALENDAR-DAY RULE is not re-instructing the same buy-in every evening: 39.29 -> 68.75 pct on instruct-or-hold. A rule that needs no memory is a rule nobody has to operate, and that is exactly why desks run one -- and it is wrong on 91.07 pct of the deadlines. b002 bizdays-mem -- THE TRAP. Identical to b003 in every respect but ONE: it counts the watching desk's Monday-to-Friday days instead of the market's settlement days. 75.0 pct against b003's 95.54. 28 deadlines wrong against b003's 5, 26 of them a real date landing EARLY by 49 days in total, and 5 premature buy-ins against b003's 2. That is the price of counting a settlement fail in the wrong kind of day, and it is a measured number rather than a warning. It is also the defect that ships, because it is right on every day both calendars agree. b003 marketcal-mem -- the strongest floor. 95.54 pct on the deadline. ⚠︎ AND THE FLOOR'S 13 WRONG CELLS SIT ON FIVE READINGS, IN TWO SHAPES. Three are readings where the keyword table read a market-wide shutdown written in counterparty language as the delivering party's fault, aged the fail a day it did not age, and called a premature buy-in with the deadline pulled early. Two are readings where the remaining extension ran past the end of the page's forward calendar and the floor abstained with NOT_DETERMINABLE; the model extended the weekly pattern past the page and got the date right. Those five readings are the entire case for spending money on this task. ⚠︎ ONE ASYMMETRY FLATTERS THE FLOORS, AND IT IS THE SAME ONE EVERY KIT IN THIS SERIES HAS: b002 and b003 share the day-counting implementation with the answer key, so they cannot misread the RULE -- they are the rule -- while the model can only read the prose. What they can and do misread is the English.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same reading (one open fail, at one scheduled run), the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

275 live calls were attempted for this kit and 269 returned something usable: 10 calibration (c000, under an 8,000-token ceiling, both hardest chains), 112 scored (r001), 41 for the missed-run ablation (a001 -- 48 readings were REPLAYED without calling, because upstream of the skip the prompts are byte-identical to r001's), and 112 for the stateless control (s001), of which 6 ran to the ceiling and came back unusable. Nothing was discarded and no run was overwritten; every result file is committed, including the free floors, which cost nothing at all. The four FREE arms -- b000, b001, b002, b003 -- plus the free missed-run floor and the whole cadence comparison add $0.00.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, AND MOST OF THEM ARE REASONING. 92.8 pct of this run's output (158,842 of 171,079 tokens) was provider-side reasoning. The prompt is 2550.5 input tokens and costs $0.00127525 of the $0.00585772 per reading; the thinking costs the other $0.00458247.
• THE CADENCE, WHICH IS THE ONLY MULTIPLIER ON THE PAGE. One run is one call per OPEN fail. Daily on this book is 120 calls a week; twice a day is 240. Halving it halves the bill and costs what the cadence lens measures -- on one skipped business day, 2 mandatory buy-ins gone permanently.
• THE RULE SHEET, REPEATED IN EVERY PROMPT. The nine buy-in rules are 3061 characters of the 10844-character prompt -- roughly 28 pct of the input, on every single reading. Moving them to a system message the provider can cache is the obvious lever and it was not measured here.
• THE FORWARD CALENDAR. Sixteen dated lines on every page so the deadline can be projected without a tool call. Shorten it and the free floor abstains more often and the model has to extrapolate; lengthen it and every reading pays.

Your volumeWhat it costs at your volume

LINEAR IN FAILS x RUNS, AND THAT IS THE WHOLE WARNING. Ten times the open fails is ten times the calls at the same cadence -- $0.00585772 a reading, so a 240-fail book read every business day is about $7.03 a week on the shared card. There is no retrieval step to amortise and no index to reuse: the population is re-read whole every run, by design, because a monitor that sliced a stream would miss the fails that did not move. The two things that do NOT grow are the carried state, which is four scalars per fail forever, and the grader, which is free.

Model choiceWhy this model

It is the tier the shared connection was already pointed at, and the kit's whole claim is that swapping is .env plus another run. No cross-model comparison was paid for and none is published; could_not_verify says so.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every stateful number on these pages: 112 readings, one completion call each, one tier. The 112-call stateless control, the 41-call missed-run ablation and the 10 calibration calls are NOT in this figure and are priced separately under Cost.cost_of_evaluation_usd. The four free floors, the free missed-run floor arm and the whole cadence comparison cost nothing at all.

The codeEvery file, and what it is for

15 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Seven of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pythe corpus generator — a swap seam

Generates 24 open fails x 5 scheduled runs = 112 snapshots from a fixed seed (SEED = 20260823), AND a second rendering of the same book with the third business day never read (89 snapshots). Every gold value is computed from the fail model through src/settlement.step, never read back off the rendered page. Each fail is given a TARGET crossing run and the generator works backwards over that market's calendar to find the intended settlement date that produces it -- because a corpus whose crossings fell where the seed put them could contain none at all on the day the ablation skips, and a missed-run measurement over a day nothing crossed measures nothing.

You change it to: PLAN, the two event pools and SEED. Keep the eight section headings and the two dated calendar blocks and every reader in the kit still parses.

# Generate the corpus and the answer key. Deterministic: one seed, one corpus, forever.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
CORPUS_MISSED = os.path.join(HERE, "data", "corpus-missed-r3")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
GOLD_MISSED = os.path.join(HERE, "data", "gold-missed-r3.jsonl")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
SEED = 20260823
RULE = "-" * 72
RUN_DATES = ["2026-03-05", "2026-03-06", "2026-03-09", "2026-03-10", "2026-03-11"]

src/marketcal.pythe market calendars — a swap seam

Four invented markets, each with its own holidays, and the two day-counting rules the kit compares: settlement days at the place of settlement, and the watching desk's own Monday-to-Friday days. This file IS the trap the kit measures. Weekends are closed in every market modelled, which is a simplification and is stated.

You change it to: MARKETS is four dictionaries of holidays. Replace them with your own markets, or with a real calendar feed -- everything downstream, including both day-counting floors, reads through settles() and business_days().

# The place-of-settlement calendars. Pure code, standard library, no market data feed.

WEEKEND = (5, 6)
MARKETS = {
MICS = tuple(sorted(MARKETS))
def d(s):
def s(x):
def holiday(mic, day):
def settles(mic, day):
def why_closed(mic, day):
def days(start, end):

src/settlement.pythe buy-in rule — a swap seam

The nine illustrative rules as arithmetic: the running count, the suspensive deduction, the forward projection of the deadline over the market calendar, the freeze once the extension period is reached, and the once-per-fail instruction. step() takes the LIST of settlement days in the window rather than a count, because when the window is wide -- the first run, or any run after a missed one -- the count can cross the extension period part-way through it and the deadline is a day INSIDE the window.

You change it to: RULE_TEXT is what the model reads and step() is what the answer key uses; they are deliberately the same nine rules in two forms. Change the extension periods, the precedence or the Rule S-5 projection convention here and the harness, the UI and all four floors move together.

# The buy-in rule as arithmetic. Pure code, integers and dates, no model and no library.

AGEING = "AGEING"
DUE_NEXT_RUN = "DUE_NEXT_RUN"
BUYIN_DUE = "BUYIN_DUE"
SUSPENDED = "SUSPENDED"
CONTEXT_INCOMPLETE = "CONTEXT_INCOMPLETE"
STATUSES = (AGEING, DUE_NEXT_RUN, BUYIN_DUE, SUSPENDED, CONTEXT_INCOMPLETE)
YES = "YES"
NO = "NO"
RAISES = (YES, NO)

src/state.pythe carried state

Four scalars between runs -- the counted age, whether a suspensive event was open, the deadline once frozen, and the previous status -- rendered as one English paragraph. Written from the arithmetic, never from the model's reply, so one bad reading cannot compound into every reading after it.

# The carried state -- the thing that makes this a monitor and not another one-shot classifier.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEFAULT_PATH = os.path.join(HERE, "data", "state.json")
def load(path=DEFAULT_PATH):
def save(store, path=DEFAULT_PATH):
def for_fail(store, fail_id):
def describe(state):

src/segment.pythe section splitter

Splits a snapshot into its eight named sections. All eight are asserted present in every document on both schedules before a run may spend.

# Split a fail snapshot into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "Fail", "Buy-in Terms", "Ageing Position",
RULE = "-" * 72
def split(text):
def render(secs):

src/select.pythe privacy gate — a swap seam

Decides which sections reach the prompt. Counterparty Desk Contact is mapped by nothing and subtracted unconditionally in the fallback, so it never leaves the machine -- personal data, and a market-sensitive fact about who cannot deliver.

You change it to: SECTION_HINTS and NEVER_SENT. Moving a section out of NEVER_SENT is the only way to widen what is sent, and check_labels.py will convict it.

# Pick which sections of a snapshot are sent. Pure code -- the last deterministic step before the

BANNER = "Synthetic Record"
FAIL = "Fail"
TERMS = "Buy-in Terms"
POSITION = "Ageing Position"
CALENDAR = "Settlement Calendar"
ACTIVITY = "Activity Log"
CONTACT = "Counterparty Desk Contact"
NOTES = "Operational Notes"
NEVER_SENT = (CONTACT,)

src/prompt.pythe prompt — a swap seam

Three parts: the fixed instruction, the carried-state paragraph, and the sections select.py let through. The middle part is the experiment; --stateless replaces it and changes nothing else, which check_labels.py asserts line by line.

You change it to: INSTRUCTION and the JSON shape. Add or drop an answered field here and in evals/scoring.FIELDS; the four are scored independently.

# Assemble the one prompt this kit sends. No template engine, no framework -- string concatenation

INSTRUCTION = """\
def build(text, carried, stateless=False):

src/watch.pythe runtime

One fail, one scheduled run, one call. Parses the arithmetic inputs off the page with regexes so the scorer can tell a reading failure from a reasoning failure, and holds the published output ceiling.

# One fail, one scheduled run, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
CORPUS_MISSED = os.path.join(HERE, "data", "corpus-missed-r3")
SYSTEM = ("You are a scheduled settlement-fail watch. You apply a written buy-in rule to one open "
MAX_TOKENS = 24000
FIELDS = ("status", "age_settlement_days", "buyin_deadline", "raise_buyin")
def corpus_dir(schedule="intact"):
def documents(schedule="intact"):
def fails(schedule="intact"):

src/adapters/__init__.pythe model adapter — a swap seam

Raw HTTP to any OpenAI-compatible endpoint, plus per-vendor adapters. No SDK, no dependency; transient statuses retried, terminal ones not.

You change it to: PROVIDER, BASE_URL, API_KEY and MODEL in .env. Adding a provider is one function and one entry in PROVIDERS; it must return token counts, because the Cost lens prices them.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/budget.pythe spend guard

Counts live calls against a cap read from the shared .env, ledgered before the call is made. Shared by every kit under the same repo root, because they share a key.

# A cap on live calls, shared by every kit on this machine. No dependency, no service.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
KIT = os.path.basename(HERE)
ROOT = os.path.dirname(os.path.dirname(HERE))
SHARED_ENV = os.path.join(ROOT, ".env")
LEDGER = os.path.join(ROOT if os.path.exists(SHARED_ENV) else HERE, ".calls-ledger.jsonl")
PAID_ARMS = ("r", "x")
FREE_ARMS = ("t", "b")
BOOT = os.urandom(6).hex()
PID = os.getpid()

evals/baseline.pythe four free floors — a swap seam

caldays, caldays-mem, bizdays-mem, marketcal-mem -- all $0.00, all scored through the identical scorer. bizdays-mem and marketcal-mem differ in exactly one thing, which calendar the days come from, so the price of the trap is a measured number.

You change it to: MODES plus one branch in review(). A fifth floor is roughly thirty lines and costs nothing to run.

# THE FREE FLOORS. Four of them, none a strawman, all of them $0.00.

MODES = ("caldays", "caldays-mem", "bizdays-mem", "marketcal-mem")
ASSUMED_EXTENSION_DAYS = 7
SUSPENSIVE_WORDS = ("depository", "market operator", "settlement engine", "blackout",
OPEN_MARKERS = ("cannot", "will not", "has not", "have not", "no stock", "holds no",
def _num(pat, text, cast=float):
def _calendar_block(text, heading):
def _log_entries(text):
def _is_suspensive(line):
def _is_open(line):

evals/scoring.pythe scorer

Exact match per cell, no model. Missed buy-ins and premature buy-ins counted apart and never averaged; the deadline scored by direction as well as by hit rate.

# Score a run against the answer key. Pure code, exact match per cell. No model grades anything.

FIELDS = ("status", "age_settlement_days", "buyin_deadline", "raise_buyin")
DATE_FIELDS = ("buyin_deadline",)
def _pct(n, d):
def _age(v):
def _norm(field, v):
def _daydiff(got, want):
def score(records, golds):
def answers_that_moved(a_cells, b_cells):

evals/cadence.pythe cadence comparison

Compares the intact schedule against one where a business day is never read, twice -- once with the model taken out entirely, once with it. Free. Prints the crossings lost and the cells that moved rather than an accuracy delta, because the accuracy delta is the small half of the damage.

# What a missed run costs -- twice, once without the model and once with it. Free -- no calls.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
PAIRS = [
def _load(run_id):
def _acc(cells, field, keys):
def main():

evals/check_labels.pythe pre-flight checks

Everything that must be true before a run may spend: the answer key replays through the same advance() the harness uses on both schedules, no snapshot restates its own carried state, the privacy guard holds and is red-proven in both directions, the two calendars actually disagree on this corpus, and a skipped run widens the calendar window and NOT the activity log.

# Everything that must be true BEFORE a run is allowed to spend. Free, and it runs in seconds.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FAILED = []
def check(name, ok, detail=""):
def load(path):
def main():

src/app.pythe local UI

http.server, no framework. Shows the carried state verbatim beside the answer and the free floor beside both, and carries a SCHEDULE SELECTOR that re-reads the same fail off the ablation corpus -- so a reader can watch a deadline move because nobody looked on Monday.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "9014"))
RECORDED = {"intact": os.environ.get("RECORDED_RUN", "r001-settle-fail"),
GOLD = {s: RUN.load_gold(s) for s in RUN.SCHEDULES}
SCHEDULE_LABEL = {"intact": "all 5 runs made",
def carried_for(doc_id, schedule):
class H(BaseHTTPRequestHandler):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2550 input and 1527 output tokens per reading (one open fail, at one scheduled run), all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

No untrusted external input surface in the shipped version. Every field that reaches the prompt comes from tools/build_corpus.py's fixed seed, including the Operational Notes, which are one of three fixed sentences. src/select.py names the notes as the field an outside party WOULD influence in a real deployment -- prose a settlements clerk types into the exceptions screen -- and sends them deliberately, so the surface is visible rather than hidden. On this corpus nobody outside the repository wrote a byte of it.

The provider key lives only in .env (gitignored) or the shared repo-root .env, read by src/config.py and never logged. config.sources() reports WHICH file a value came from and never the value. config.save() opens the file 0600 before the write rather than chmod-ing it after, so the key never exists at the default umask even for an instant. src/app.py redacts the key and the base URL out of any provider error before it reaches the browser, and tools/shoot_ui.mjs blanks the key entirely and refuses to start if something is already listening on the port, so no screenshot can spend.

The experimentWe did NOT attack it — and this is the surface we left open

An indirect prompt injection needs a field somebody outside can write into, and this kit has exactly one: the Operational Notes, prose a settlements clerk types into the exceptions screen. It is SENT, on purpose, and one of the three shipped sentences is instruction-shaped for exactly that reason -- "The counterparty's settlements desk has asked to be told before any buy-in is instructed on this line". On this corpus that sentence comes from a seeded generator, so there is nothing adversarial in it to catch and no trial is claimed. A version pointed at real clerk prose reopens the question and should be attacked before it ships. No attack was fired at this kit and none is claimed. The three boundaries below were checked in code on 2026-08-23, and only the first of them is red-proven in both directions rather than argued from an absent code path.

The first row is the only one measured in both directions. The other two are guarantees about what is ABSENT from the code, and an absent path is checked by grepping for the names we know -- a path called something else would pass.

The result0 attack trials, three boundaries checked -- and the privacy boundary red-proven by removing the guard and watching all 112 documents leak.

The Operational Notes ARE the field an outside party would influence in a real deployment, and this kit sends them rather than hiding them -- one of the three shipped sentences is even instruction-shaped, chosen for exactly that reason. But on this corpus they are one of three sentences from a seeded generator, so there is nothing adversarial in them to catch. A version pointed at real clerk prose reopens the question and should be attacked before it ships. What IS measured is the privacy boundary, in both directions, before every run.

HonestyWhat this does not prove

• Whether a real deployment's Operational Notes -- prose a settlements clerk writes freely -- would carry an instruction the model follows. Not applicable to the shipped corpus, and the first thing to attack if this is pointed at a real book. The shipped note asking to be told before any buy-in is instructed is the shape to probe.
• Whether the privacy guard covers anything beyond the section it names. It subtracts Counterparty Desk Contact unconditionally, and it is red-proven in both directions -- but a counterparty's identity also appears as a reference in the Fail section, which IS sent. A reference is not a name; whether it is re-identifiable against a real book is not something this repository can test.
• Whether the state file is safe under concurrency. src/state.save() replaces atomically, which is correct for one writer and is not a concurrency model; two watches on one book have never been run and would race. On a RUNNING COUNT a lost run is a permanently wrong deadline rather than a missing row.
• Whether a truncated reply can ever be partially trusted. The stateless control lost 6 calls to the output ceiling and src/watch._parse rejected all of them outright; nothing here has looked at whether a cut-off reply's first field was usable.
• Whether the absence checks catch a path named something else. evals/check_labels.py greps for eight names it knows. A function called send_to_agent would pass it.
• What the provider retains. The prompt carries no personal data by construction, but what happens to a request after it leaves this machine is outside this repository.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

No buy-in instruction, non-configurable. This kit produces a watchlist status, a counted age, a deadline date and an instruct-or-hold call for a settlements desk to validate. It never instructs a buy-in agent, places an order, cancels a trade or releases cash, and there is no setting that makes it.

Everywhere and nowhere -- it is a property of what is ABSENT. The only writers in the kit are evals/run.py (results/*.json) and src/state.save (data/state.json). src/watch.py states the rule where the calls are made rather than only on a page.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• IT IS NOT A CHECK ON WHETHER THE ANSWER IS RIGHT. The arithmetic is correct whatever the model says, which means a wrong reading is a wrong watchlist row -- and on this task a wrong watchlist row can read INSTRUCT THE BUY-IN NOW with a rationale that cites two rules by name.
• IT IS NOT A DEFENCE AGAINST A MISSED RUN. The guardrails are about what the kit will not do. The most expensive failure measured on this page is something nobody did at all: 2 mandatory buy-ins worth $8,315,720.00 vanished from the record because one scheduled run was not made.
• IT IS NOT AN INJECTION DEFENCE. The Operational Notes are attacker-writable in any real deployment and are sent verbatim, deliberately, so the surface is visible. Nothing here has been attacked.

WatchedWhat is watched, and why that one

The kit measures 35 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

16 measured by the latest run 19 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (cadence every business day, 18:00, at the watching desk, fails 24, stateless False) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

10 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

triage · no model in the path — a baseline, not a peer column — 5 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 5 runs measured the same system — they differ on context_incomplete_cells, crossing_cells, documents, floor, memory_cells, quiet_cells, readings_scored, run_skipped, runs_missed_detected, schedule, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

triage · with the model in the path — 4 runs. Columns here are only ever compared with each other.

not a time series No two of these 4 runs measured the same system — they differ on context_incomplete_cells, crossing_cells, documents, fails, max_tokens, memory_cells, quiet_cells, readings_scored, run_skipped, runs_missed_detected, schedule, stateless — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 16 chips that all say so.

DeviationsWhat deviated

0 breaches across 10 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-run evals/check_labels.py (free, under a second) on any change to tools/build_corpus.py, src/settlement.py, src/marketcal.py, src/segment.py or src/select.py -- it refuses to let a run spend if the answer key stops replaying, if a snapshot starts restating its own carried state, if the privacy guard stops holding, or if the two calendars stop disagreeing on this corpus. Re-run evals/cadence.py (also free) on any change to the schedule.

In one lineNo framework, deliberately

No framework and no dependencies at all -- json, os, re, random, datetime, argparse, urllib, http.server and concurrent.futures from the standard library. requirements.txt names nothing and the emptiness is load-bearing: a forker runs this on whichever key they already hold, without installing a client for a vendor they will never call.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph, and the one place it looks like one is a straight line. Each fail is a chain of up to five readings -- one per business day -- with no branching and no retries beyond the transport layer. Different fails are independent, which is where the concurrency is. The only edge that matters is the one carrying four scalars from one reading to the next, and it carries them through pure arithmetic rather than through the model.

The other sideWhat a framework costs you

• No scheduler. This is a watch that carries state and is still INVOKED, not woken -- and the cadence is the single most important thing about it. That is the honest cost of shipping no framework, and the kit narrows it by DETECTING a missed run rather than by preventing one.
• No calendar service. Four invented markets in a Python dict. A production book settles in dozens of markets whose holidays change, and a stale calendar produces exactly the defect b002 measures.
• No persistence beyond a JSON file. data/state.json is one writer, replaced atomically, and nothing here has been tested with two.
• No retry policy above the transport. A reading that errors advances the count anyway -- deliberately, so one transport error does not turn into four scored failures -- but nothing re-attempts it.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-settle-fail on the same tier, one business day never read (THE ABLATION), 2026-08-23. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 8 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

5 runs not plotted. a000-settle-fail-missedrun-floor, b000-settle-fail-caldays, b001-settle-fail-caldaysmem, b002-settle-fail-bizdaysmem, b003-settle-fail-marketcalmem recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the population is re-read whole on each scheduled run, and the prior run's carried state is what makes a reading a CHANGE.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved; every field was written by the generator committed beside it. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe status, the counted age, the buy-in deadline date and the instruct-or-hold call, per reading, exact match against the computed key

For each of the 112 readings and each of the four answered fields, did the reply equal the value tools/build_corpus.py computed from the fail model.

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model. The same function all four free floors, the stateless control and the missed-run ablation are scored through, so the seven arms are comparable by construction.

Every grader on these pages scored the same 112 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 112 per field. A reading whose reply did not parse counts as a MISS in every field rather than being dropped -- a run that lost readings must not score better for having lost them.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl and data/gold-missed-r3.jsonl, computed by tools/build_corpus.py from the fail model and the market calendars at generation time and re-derived through evals/run.advance by evals/check_labels.py before any run may spend.

How tight can the band be? No tuned threshold anywhere on the model path -- the reply decides, and the scorer compares strings and integers. The one threshold in the kit is Rule S-9's DUE_NEXT_RUN band, and it is not tuned: it IS the cadence.

Cadence: Every run, automatically -- evals/run.py scores in the same process that made the calls, so no arm can exist without its grade. evals/check_labels.py runs BEFORE any run may spend and re-derives the whole key on both schedules; re-run it free on any change to the generator, the rule, the calendars, the splitter or the privacy gate.

The decisionWhen to reach for it

In one lineContext-incomplete recall -- the guardrail, scored

Of the 10 readings whose agreement carries no extension period or whose feed captured no intended settlement date, how many were reported CONTEXT_INCOMPLETE rather than aged against a guessed number.

MethodHow the test was run

evals/scoring.py::score, the same pass as the cell match.

Every grader on these pages scored the same 112 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

recall = correctly abstaining readings / 10

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the 10 readings whose gold status is CONTEXT_INCOMPLETE, from data/gold.jsonl.

How tight can the band be? No threshold. The rule is categorical: an absent value is an abstention.

Cadence: Every run, in the same pass as the cell match. It is a rate over the 10 readings whose gold status is CONTEXT_INCOMPLETE, so it is only meaningful while the corpus keeps carrying them -- and evals/check_labels.py does not assert that it does, which is stated here rather than assumed.

The decisionWhen to reach for it

In one lineWhat a missed run costs, in crossings rather than in points

How many mandatory buy-ins fell due on the day nobody looked, how many of those were merely late, and how many are gone permanently because the fail settled in full before the next run.

MethodHow the test was run

tools/build_corpus.py writes the counts; evals/cadence.py prints them.

Every grader on these pages scored the same 112 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

Read off the two answer keys: crossings on the skipped run, minus those whose fail still appears on a later reading of the ablation schedule. No model, no call.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl against data/gold-missed-r3.jsonl -- the same truth, with the run that must instruct each buy-in moved.

How tight can the band be? No threshold. A crossing is lost when the fail has no later reading on the ablation schedule, which is a set membership test.

Cadence: On demand and free: python3 -m evals.cadence. It reads the committed result files, so it is re-runnable forever without a key. Re-run it whenever the schedule, the corpus or the ablation changes -- and note that the crossings half needs no result file at all: it comes from the two answer keys, which tools/build_corpus.py regenerates from the seed.

The decisionWhen to reach for it