The business caseThe problem this solves

A stuck-order queue tells you how late every task is. It does not tell you which orders are going to MISS. Those are two different questions, and the gap between them is where the working days go: an order whose supplier quoted a fifteen-day lead time this morning is already lost, and nothing on its task table is late yet. By the time the aging thresholds catch up, the committed date is days away and the customer has been told the wrong thing twice. The evidence that would have said so on day two is sitting in the activity notes, in a sentence somebody typed at five o'clock, next to four other sentences that mention a number of working days and commit to nothing. A fulfilment coordinator opening every stalled order in the queue each morning, reading the task ages against the jeopardy thresholds, then reading the activity notes to find the supplier quote or rebooked date that means the committed date is already unreachable -- and remembering which of them were raised yesterday.

The inputThe actual order snapshots

The corpus is 240 order snapshots, 0.72 MB (jsonl 1 · txt 240). A real stuck-order queue names an operator's customers, their site addresses, their mobile numbers, a supplier's commercial lead times and the engineer who did not turn up. There is no public corpus of (order snapshot, activity log, adjudicated jeopardy verdict) for the same reason there is no public corpus of production alert histories. Generating it also bought the one thing this kit could not do without: the OUTCOME of every order is known at generation time and appears on no page. You cannot ask 'how many working days before the miss did this pack surface it' of a corpus that does not know which orders missed.

Synthetic Record
----------------------------------------------------------------
This file is synthetic. It was generated for an open evaluation kit and describes
no real operator, customer, supplier, order or person. Order ORJ-0001, surveillance day D1.

Order
----------------------------------------------------------------
Order reference     : ORJ-0001
Product             : Site-to-site connectivity, dual circuit
Segment             : Public sector framework
Install type        : New build, new duct required
Order placed        : Mon 16 Feb 2026
Committed date      : Mon 09 Mar 2026

Jeopardy Policy
----------------------------------------------------------------
Overdue threshold   : 2 working days
Blocker threshold   : 4 working days
Rules stated in the fulfilment jeopardy policy:
  1. An order with an open task MORE than the overdue threshold past its due date
     is reported JEOPARDY, under MILESTONE_OVERDUE.
  2. An order with an open task HELD MORE than the blocker threshold is reported
     JEOPARDY, under BLOCKER_AGED.
  3. An order whose Activity Log states, on this day, a lead time, a rebooked
     appointment, a consent window or a booked date that reaches BEYOND the
     committed date -- stated either as a number of working days ahead of today,
     or as a date -- is reported JEOPARDY, under LEADTIME_EXCEEDS_REMAINING,
     whether or not any task is yet overdue. A commitment landing ON the
     committed date does not engage this rule. Nor does a note that states a
     number of working days without committing to when anything will be ready --
     a retention rule, a cooling-off period, an escalation service level, a
     window that has already closed.
  4. An order reported JEOPARDY on the previous surveillance day and STILL

The outcomeWhat a good result looks like

Per order per surveillance day: the status to REPORT (CLEAR, WATCH or JEOPARDY), which of the five policy rules was applied, and the open task most responsible -- plus a four-field carried state the next day is judged against, written by code from the parsed task ages and never from the model's reply. Across the window: the day each order was first raised, and the working days that left before the committed date.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• A queue whose jeopardy rules are all about task ages against thresholds — the free aging floor, at $0.00
  It applies rules 1, 2 and 5 perfectly and names the top blocker on 240 of 240. It is what your fallout report already does, and this kit measured that it is not improved on by anything.
• A rule about a commitment written into an activity note -- a supplier lead time, a rebooked slot, a consent window — the fast tier, with the carried state
  This is the whole gap. The aging floor misses 9 of the 30 slipped orders outright and loses 45 of the 120 available lead days, all of it on the two trajectories where nothing ever goes overdue. The model wins every one of those days back.
• Notes written to a fixed template by a system rather than by a person — a pattern, at $0.00
  That is exactly what this corpus is, and the pattern arm scores 100 pct on every axis of it. If your activity log is machine-written -- a status feed, a supplier API, a fixed set of dispositions -- write the pattern and keep the money.
• Telling a NEW jeopardy from one that has been running for days — free code, with memory
  Rule 4 is memory and memory is arithmetic. The floor with memory names the rule correctly on 86.25 pct against 69.58 pct without it, and moves the lead time by exactly zero.
• Deciding whether to rebook an engineer or tell a customer their date has moved — a person, reading the status and the rule the pack named
  The two content errors in this run were both the model reading a narrative note as a change of state, and one of the two did not reproduce on a second ask. Nothing here notifies anybody and nothing should be added that does without a human step in front of it.

And where nothing here is good enough:

• An order book of thousands running every working day — nothing here yet -- measure it first
  The cost is per order per surveillance day. Five thousand open orders on a daily cadence is 25,000 calls a week, about $112 a week on the shared rate card, and the free floor answers most of them identically. The question is which orders are worth pointing a model at, and this kit has not answered it.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-23. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-23 — r001-order-jeopardy. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with NO key configured reproduces the corpus byte-identically (python3 tools/build_corpus.py), passes every pre-flight assertion (python3 -m evals.check_labels), scores all three free floors (python3 -m evals.run --run-id b002-order-jeopardy-regex --baseline regex, and the other two), re-scores the paraphrase probe's free half (python3 -m evals.paraphrase --free) and serves the whole board in a browser (python3 -m src.app, port 9005). Six commands, plain python3, no install step -- requirements.txt names nothing because the kit imports nothing outside the standard library. What it cannot do without a key is re-run the scored eval or the model's half of the paraphrase probe.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per surveillance day: one HTTP request carrying the assembled prompt, and the reply parsed to a status, a rule and a top blocker. Splitting the snapshot into sections, dropping the one no field asks for, parsing the task table and the days remaining, and advancing the carried state all happen outside this clock and cost no network at all. p95 (24244 ms) is 2.7 times p50 (8972 ms), and the slowest single call took 62480 ms. Output length tracks how hard the READING is, not how long the page is: every snapshot is between 2,941 and 3,482 bytes and replies ran from 222 to 7096 output tokens.

Current processWhat it replaces

A fulfilment coordinator opening every stalled order in the queue each morning, reading the task ages against the jeopardy thresholds, then reading the activity notes to find the supplier quote or rebooked date that means the committed date is already unreachable -- and remembering which of them were raised yesterday.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptArchitectureArchitectureApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• 3 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one orders (5 surveillance days each) is

No split, and no chunking. The unit is an ORDER -- a run of five consecutive surveillance days processed strictly in order, because day 3's prompt contains state produced by day 2. Each snapshot goes whole into one call, minus the one section no field asks for.

SetupWhat the setup figure measured

There is no index and no retrieval step. tools/build_corpus.py writes 240 documents and the answer key from a fixed seed with no clock read and no model called; nothing is embedded, ranked or cached.

LicenceLicence

MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved; every order, product, task, supplier commitment, activity note and name is invented here. Every contact number is in the 07700 900000-900999 range reserved by Ofcom for fiction and never allocated to a subscriber.

Bring your ownBring your own order snapshots

Replace src/jeopardy.py FIRST. It is your operator's policy, not a utility, and data/gold.jsonl is literally its output -- change the rules and re-run tools/build_corpus.py and the answer key follows. Change the policy TEXT in the generator in the same edit, because the model reads the rules off the page rather than out of the prompt. Then point the generator at your own task families, or delete it and drop real snapshots into data/corpus/ named <ORDER>-D<n>.txt with the same seven section headings.

⚠︎ And what stops being true when you do: Every figure on this page is a property of a 48-order corpus with eight planted trajectories, five surveillance days each and an activity log written from sixteen templates, measured once. Two numbers will not survive contact with a real book. The lead times are a property of the TRAJECTORY MIX -- six orders per trajectory was a design decision, so a book with more slow-aging orders and fewer supplier-quote failures moves the headline without anything having changed about the arms. And the free pattern's 100 pct is a property of the WORDING: it reads 1 of the 12 rule-3 commitments once they are rephrased.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 4,674 characters assembled, 1,695 are instructions — 36% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Replayed from the kit's own src/prompt.py::build on ORJ-0007-D2 with the exact carried state r001-order-jeopardy recorded for that call ({"prev_status": "CLEAR", "prev_rule": "NONE", "days_in_jeopardy": 0, "first_raised_day": null}) -- the identical code path the run used, not a paraphrase. THE SPLIT IS IN CHARACTERS AND THE PAGE SAYS SO: per-part TOKEN counts were not measured, because measuring them means sending nested prefixes of the prompt to the provider and this kit spent no calls on it. The billed totals (273043 in / 312941 out over 240 calls) are measured; nothing here apportions them across the parts.

Full promptThe prompt in full

You apply a written fulfilment jeopardy policy to one open order on one surveillance day. You answer with one JSON object and no other text.

You are reviewing one open telecommunications order on one surveillance day, against the written
fulfilment jeopardy policy reproduced in the snapshot below.

Apply the policy exactly as written. Rule 4 depends on what was reported on EARLIER surveillance
days, which you cannot see; what is known about this order's history is stated under "Carried
state" and is the only history available to you. Do not assume anything about earlier days beyond
it.

Read the Activity Log as carefully as the Open Tasks table. Rule 3 is about a forward commitment
stated in the notes — a supplier lead time, a rebooked appointment, a statutory consent window —
measured against the working days that remain. A note that merely mentions a number of working
days, such as a retention period, a cooling-off window or an escalation service level, is not a
forward commitment and does not engage rule 3.

Answer with a single JSON object and nothing else:

{"status": "CLEAR|WATCH|JEOPARDY",
 "rule": "NONE|MILESTONE_OVERDUE|BLOCKER_AGED|LEADTIME_EXCEEDS_REMAINING|ESCALATED_SUSTAINED",
 "top_blocker": "the open task most responsible, named exactly as it is written in Open Tasks",
 "rationale": "one sentence, naming the rule or the threshold you applied"}

"status" is what to report for this order today. "rule" names which policy rule you applied, or
NONE. "top_blocker" is the open task with the most working days past due or held; if no task is
past due or held, it is the open task nearest to its due date; if the order has no open tasks at
all, it is the word None.

Carried state
----------------------------------------------------------------
On the previous surveillance day this order was reported CLEAR. It has not been in jeopardy on any earlier day of this window.

Order snapshot
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
This file is synthetic. It was generated for an open evaluation kit and describes
no real operator, customer, supplier, order or person. Order ORJ-0007, surveillance day D2.

Order
----------------------------------------------------------------
Order reference     : ORJ-0007
Product             : Ethernet leased line, 100 Mbit/s
Segment             : Small business, single site
Install type        : Migration from a third-party circuit
Order placed        : Mon 02 Feb 2026
Committed date      : Mon 09 Mar 2026

Jeopardy Policy
----------------------------------------------------------------
Overdue threshold   : 2 working days
Blocker threshold   : 4 working days
Rules stated in the fulfilment jeopardy policy:
  1. An order with an open task MORE than the overdue threshold past its due date
     is reported JEOPARDY, under MILESTONE_OVERDUE.
  2. An order with an open task HELD MORE than the blocker threshold is reported
     JEOPARDY, under BLOCKER_AGED.
  3. An order whose Activity Log states, on this day, a lead time, a rebooked
     appointment, a consent window or a booked date that reaches BEYOND the
     committed date -- stated either as a number of working days ahead of today,
     or as a date -- is reported JEOPARDY, under LEADTIME_EXCEEDS_REMAINING,
     whether or not any task is yet overdue. A commitment landing ON the
     committed date does not engage this rule. Nor does a note that states a
     number of working days without committing to when anything will be ready --
     a retention rule, a cooling-off period, an escalation service level, a
     window that has already closed.
  4. An order reported JEOPARDY on the previous surveillance day and STILL
     breaching any of rules 1 to 3 today is reported JEOPARDY again, under
     ESCALATED_SUSTAINED, in place of the rule it is still breaching.
  5. An order with an open task past its due date, or held, but within both
     thresholds is reported WATCH. An order with neither is reported CLEAR. A task
     that has closed stops counting and is not listed.

Surveillance Day
----------------------------------------------------------------
Day                            : D2
Date                           : Tue 03 Mar 2026
Working days to committed date : 4

Open Tasks
----------------------------------------------------------------
Open tasks on this order:
  Fibre splice                       Field engineering          due in 3 working days

Activity Log
----------------------------------------------------------------
Notes recorded against this order since the previous surveillance day:
  - The gang that covers fibre splice is booked for Tue 10 Mar 2026; there is no earlier slot.
  - Porting desk validated the account details against the losing carrier's record.

Raw responseThe raw response

{"status": "JEOPARDY", "rule": "LEADTIME_EXCEEDS_REMAINING", "top_blocker": "Fibre splice", "rationale": "Activity Log shows the fibre splice gang is booked for Tue 10 Mar 2026, beyond the committed date of Mon 09 Mar 2026, so rule 3 applies."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch a telecom order before it misses its date — 240 order snapshots. One model answered, and every answer was then graded Four different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

There is no LLM-as-judge in this kit and that is not a shortcut. Every field it answers is a closed set -- three statuses, five rule codes, and a task name copied from a table on the page -- so correctness is a matter of equality, not of reading. And the HEADLINE is not a comparison of answers at all: it is a replay of each arm's own verdicts to find the day it first raised each order, which is arithmetic over a column.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran all 276 calls is kept out of these tables per this estate's naming rule, so nothing here is what was paid. The real spend for this kit's whole build is recorded in the shared call ledger, not on this page.

Grading unitWhat the grading figure prices

Grading calls nothing. A measured $0.00 -- and three of the four arms on every board here are pure code, so most of what this page compares cost nothing to produce either.

The gradersFour ways to grade

⚑ THERE ARE THREE FREE FLOORS AND THE ONE ABOVE IS THE WEAKEST OF THEM. Read all three before reading the model's row. The aging floor catches 21 of the 30 slipped orders and wins 75 of the 120 available lead days. Adding MEMORY changes neither number by one -- rule 4 renames a jeopardy that is already raised and cannot raise one -- so memory is worth exactly 16.67 points of RULE accuracy and nothing else. Adding a PATTERN over the Activity Log takes it to 30 of 30 and all 120 lead days, at 100 pct on every field, for $0.00: BETTER than the model, which scores 98.75 pct on status. ⚠︎ That last row is why evals/paraphrase.py exists. Rewrite the same commitments in words the generator never produced and the pattern falls to 54.17 pct while the model holds at 100.0 pct. The free arm's perfection is a property of sixteen sentence templates written by the author of the pattern that reads them.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

⚑ 276 LIVE CALLS WERE MADE FOR THIS KIT AND EVERY ONE OF THEM IS BEHIND A FIGURE ON THIS PAGE. 240 were the scored run, 10 were the calibration that set the output ceiling, 24 were the model's half of the paraphrase probe, and 2 were the UI screenshot -- one aimed at a failure that did not reproduce, and one retaken after the shot was renamed to say so. NOTHING WAS DISCARDED and no run was fired twice. Everything else on this page -- three free floors, the free half of the paraphrase probe, the scorer, the pre-flight and the whole board in the UI -- is pure code and costs $0.00. Figures are projected onto the Google Gemini 3 Flash card, not the real spend.

Cost driversWhat actually moves the bill

• PROVIDER-SIDE REASONING, 94.87 pct of output tokens (296887 of 312941) and therefore most of the bill. It was left at the provider's default and this kit has not measured what turning it off does to the answers.
• HOW HARD THE READING IS, NOT HOW LONG THE PAGE IS. Every snapshot is between 2,941 and 3,482 bytes and output tokens ran from 222 to 12000 -- a 54-fold spread on a corpus of one shape. ⚠︎ AND THE SPREAD IS NOT EXPLAINED BY DIFFICULTY: the single call that reached the ceiling was one of the quietest snapshots in the set.
• THE FIXED HEAD OF THE PROMPT. The system message, the instruction and the five policy rules are 3237 of 4847 characters -- 67 pct, byte-identical on all 240 calls except for two threshold numbers. The order's own situation is the smaller half of what is sent.
• ⚑ AND THE LARGEST COST LEVER ON THIS PAGE IS NOT A TOKEN SETTING. It is WHICH ORDERS GET SENT. The free aging floor answers 86.25 pct of snapshots identically to the model for nothing; the whole measured advantage sits on the two trajectories where nothing is overdue. An operator who ran free code first and sent a model only the orders it calls CLEAR would pay for a fraction of the book -- unmeasured here, and the obvious next experiment.

Your volumeWhat it costs at your volume

Linear in ORDER-DAYS, and flat in window length, which is the unusual half. Each surveillance day is one call whose input is one snapshot plus four carried fields, so day 300 of an order costs exactly what day 3 costs. 240 snapshots project to $1.0753 on the shared rate card, so ten times the set is about $10.75. ⚠︎ THE FIGURE THAT MATTERS IS NOT 10x THE CORPUS, IT IS THE BOOK: a desk with five thousand open orders on a daily cadence is 25,000 calls a week, about $112 a week. What does NOT scale is wall clock -- days within one order are strictly serial, so a long window is a long queue no width can shorten.

Model choiceWhy this model

It is the tier the shared key is pointed at, and it is the cheap one -- the right place to start a question whose honest answer turned out to be 'free code already does this on this corpus', which here it does.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every model figure on these pages: 240 surveillance days, one completion call each, one tier. The 10 calibration calls and the 24 paraphrase-probe calls are accounted for separately in Cost.cost_of_evaluation_usd and are not projected here. The three free floors made no calls at all.

The codeEvery file, and what it is for

13 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pythe corpus generator — a swap seam

Generates 48 orders x 5 surveillance days = 240 snapshots from a fixed seed (SEED = 20260823) across twelve task families, six products and eight planted trajectories -- five that slip their committed date and three that deliver. It writes the outcome of every order into the answer key and onto no page, which is what makes this a replay rather than a classification set. The gold labels are src/jeopardy.replay's output, never typed.

You change it to: Point it at your own task families and products, or delete it and drop real snapshots into data/corpus/ named <ORDER>-D<n>.txt with the same seven section headings. Everything downstream reads by order and day and does not care where the documents came from.

# Generate the synthetic order book: 48 orders x 5 surveillance days = 240 snapshots, plus the

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
SEED = 20260823
ORDERS = 48
DAYS = 5
RULE = "-" * 64
TASKS = [
PRODUCTS = [

src/jeopardy.pythe jeopardy policy — a swap seam

The written policy as pure code: two thresholds, five rules, three statuses. Rules 1, 2 and 5 are arithmetic over task ages; rule 3 is a forward commitment stated in prose and this module never reads it -- it is passed in. Rule 4 is memory. The branch order is load-bearing: rule 4 is tested FIRST, so a running jeopardy is reported as sustained rather than as if it had just arrived.

You change it to: step() is your operator's policy. Change it there and re-run tools/build_corpus.py, which recomputes the whole answer key from the same function -- the gold cannot drift from the rules because it is their output. Change the printed text in the generator to match, because the model reads the policy off the page and not out of the prompt.

# The fulfilment jeopardy policy, as arithmetic. Pure code, no model.

CLEAR, WATCH, JEOPARDY = "CLEAR", "WATCH", "JEOPARDY"
NONE = "NONE"
MILESTONE_OVERDUE = "MILESTONE_OVERDUE"
BLOCKER_AGED = "BLOCKER_AGED"
LEADTIME_EXCEEDS_REMAINING = "LEADTIME_EXCEEDS_REMAINING"
ESCALATED_SUSTAINED = "ESCALATED_SUSTAINED"
STATUSES = (CLEAR, WATCH, JEOPARDY)
RULES = (NONE, MILESTONE_OVERDUE, BLOCKER_AGED, LEADTIME_EXCEEDS_REMAINING, ESCALATED_SUSTAINED)
def initial_state():

src/state.pythe carried state — a swap seam

SEAM 2 -- the thing that makes this a monitor. Four fields per order (the status reported yesterday, the rule applied, consecutive days in jeopardy, the day it was first raised), written from the arithmetic and never from the model's reply, and rendered into one English sentence for the prompt. It costs the same on day 300 as on day 3.

You change it to: load()/save() are a JSON file today. Point them at a table, a key-value store or the order management system itself and nothing else changes -- for_unit() and advance() are the whole interface, and describe() is the only thing the prompt sees.

# The carried state — what makes this a monitor rather than another one-shot classifier.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEFAULT_PATH = os.path.join(HERE, "data", "state.json")
def load(path=DEFAULT_PATH):
def save(store, path=DEFAULT_PATH):
def for_unit(store, order_id):
def advance(store, order_id, max_past_due, longest_held, overdue_at, blocker_at, lead_excess, day):
def describe(state):

src/segment.pythe section splitter

Splits a snapshot into its seven named sections. Pure code; evals/check_labels.py asserts all seven are present in all 240 documents before a run may spend.

# Split an order snapshot into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "Order", "Jeopardy Policy", "Surveillance Day",
RULE = "-" * 64
def split(text):
def render(secs):

src/select.pythe section selector — a swap seam

Decides which sections reach the model. Customer Contact is mapped by no field and is subtracted unconditionally by _fallback(), so the one section whose subject is a PERSON never leaves the machine even when a hint matches nothing.

You change it to: SECTION_HINTS maps a field to the sections it needs; NEVER_SENT names what is withheld whatever happens. Add a section your snapshots carry and decide, once, which of the two lists it belongs in.

# Pick which sections of an order snapshot are sent. Pure code — the last deterministic step

BANNER = "Synthetic Record"
ORDER = "Order"
POLICY = "Jeopardy Policy"
DAY = "Surveillance Day"
TASKS = "Open Tasks"
CONTACT = "Customer Contact"
LOG = "Activity Log"
NEVER_SENT = (CONTACT,)
SECTION_HINTS = {

src/monitor.pythe fact parser

Reads the two thresholds, the days remaining and every open task's age off the page with a regex -- and DELIBERATELY DOES NOT READ THE ACTIVITY LOG. That asymmetry is the kit: rule 3 lives in prose, and a runtime that could parse it would make the model redundant without measuring anything. evals/check_labels.py asserts the facts this module exposes carry no rule-3 field.

# One surveillance day, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = ("You apply a written fulfilment jeopardy policy to one open order on one surveillance "
MAX_TOKENS = 12000
FIELDS = ("status", "rule", "top_blocker")
CALIBRATION_ORDERS = ("ORJ-0007", "ORJ-0016", "ORJ-0024", "ORJ-0032")
def documents():
def units():
def load_doc(doc_id):

src/prompt.pythe prompt

Three parts: the fixed instruction, the carried-state sentence, and the selected sections in document order. The policy is NOT restated in the instruction -- it is printed on every snapshot, so a forker changes it in one place and the answer key follows.

# Assemble the one prompt this kit sends. No template engine, no framework — string concatenation

INSTRUCTION = """\
def build(text, carried, stateless=False):

src/adapters/__init__.pythe model adapter — a swap seam

SEAM 1 -- the model. Raw HTTP for every provider, standard library only. Returns input/output tokens, reasoning tokens and finish_reason beside the text, and separates a transport failure from an HTTP status so a dropped connection is retried rather than recorded as a lost document.

You change it to: .env plus the same run again. Any OpenAI-compatible provider or Anthropic.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

evals/baseline.pythe three free floors

aging (task ages, no memory, never opens the Activity Log), agingmem (the same, told yesterday's status) and regex (the same, plus a genuine pattern over the Activity Log in both the count and the date form). Each differs from the one before it by exactly one thing, so the gaps price memory and prose-reading separately. 0 calls, $0.00, scored through the identical scorer.

# Three free floors, none of them a straw man. No key, no calls, $0.00.

WORDS = {"one": 1, "two": 2, "three": 3, "four": 4, "five": 5, "six": 6, "seven": 7, "eight": 8,
NUM = r"(\d+|" + "|".join(WORDS) + r")"
FORWARD_PATTERNS = [
FORWARD_RE = [re.compile(p, re.I) for p in FORWARD_PATTERNS]
DATE = r"((?:Mon|Tue|Wed|Thu|Fri|Sat|Sun) \d{2} [A-Z][a-z]{2} \d{4})"
FORWARD_DATE_PATTERNS = [
FORWARD_DATE_RE = [re.compile(p, re.I) for p in FORWARD_DATE_PATTERNS]
COMMITTED_RE = re.compile(r"^Committed date      : (.+)$", re.M)
def _value(tok):

evals/scoring.pythe lead-time scorer

Replays each arm's OWN answers order by order to find the day it first said JEOPARDY, and turns that into working days before the committed date. The per-cell accuracy is computed in the same pass and reported beside it, never instead of it -- an arm can be wrong on four snapshots and still have surfaced the order eight days early.

# Score a run against the computed answer key. Pure code, no model, no judge.

STATUSES = ("CLEAR", "WATCH", "JEOPARDY")
RULES = ("NONE", "MILESTONE_OVERDUE", "BLOCKER_AGED", "LEADTIME_EXCEEDS_REMAINING",
def _pct(n, d):
def _norm(v):
def _median(xs):
def score(records, golds):
def _by_pattern(leads):
def compare(arms):

evals/paraphrase.pythe paraphrase probe

Rewrites 24 forward commitments into phrasings that appear nowhere in the generator and that no pattern in evals/baseline.py matches, stating the identical fact, and re-scores both arms. It exists because the free pattern scores 100 pct on the corpus as generated, and publishing that number without testing it would be publishing an artefact of the corpus.

# The paraphrase probe: does an arm read the SENTENCE, or the PATTERN?

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
COUNT_FORMS = [
DATE_FORMS = [
def _pick(s):
def _forward_note(g):
def pick(golds, n_each=12):
def rewrite(text, note, form, remaining, committed, day_offset_date):

evals/check_labels.pythe pre-flight

Everything that must be true before a run may spend: 240 documents, 48 gap-free orders, gold reproducible from jeopardy.replay, the parser agreeing with the key on every arithmetic fact, the privacy guard red-proven in both directions, and eight corpus-honesty assertions each of which measured a non-zero the day it was written.

# Everything that must be true of the corpus BEFORE a run may spend. Free, no key, no model.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
FAIL = []
DOCS = 240
ORDERS = 48
DAYS = 5
def check(name, ok, detail=""):
def main():
WORDS = {"one": 1, "two": 2, "three": 3, "four": 4, "five": 5, "six": 6, "seven": 7, "eight": 8,

src/app.pythe local UI

Port 9005. Replays any order through the policy and the three free floors with NO KEY AND NO CALLS -- three of the four rows on the board are pure code -- and adds the model's own row on one click when a key is configured. No route in the file reassigns, rebooks, contacts or writes anything.

# The minimal local UI. Standard library only -- python -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
PORT = int(os.environ.get("PORT", "9005"))
def gold():
def replay_free(order_id):
class H(BaseHTTPRequestHandler):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 1137 input and 1303 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

No untrusted external input surface in the shipped version. Every field that reaches the prompt comes from tools/build_corpus.py's fixed seed, including the Activity Log, which is drawn from sixteen sentence templates. src/select.py names the Activity Log as the field an outside party WOULD control in a real deployment -- engineers, suppliers' agents and desk staff write it -- and sends it deliberately, because rule 3 lives there. The surface is visible rather than hidden; on this corpus nobody outside the repository wrote a byte of it.

The provider key lives only in .env (gitignored) or the shared repo-root .env, read by src/config.py and never logged. config.sources() reports WHICH file a value came from and never the value. config.save() opens the file 0600 before the write rather than chmod-ing it after. src/app.py redacts the key and the base URL out of any provider error before returning it to the browser.

The experimentWe did not attack it -- and the boundary that matters most was proven by breaking it on purpose

An indirect prompt injection needs a field an outside party controls that reaches the prompt, and this corpus has none. So the five rows below are boundaries, not payloads. Two of them were red-proven rather than asserted: the privacy guard was replaced with the naive fallback every sibling kit once shipped and all 240 snapshots leaked the customer's name and mobile number, then it was restored and all 240 held. A guard that has never been made to fail is a guard nobody has tested. Confirmed by assertion and by reading the recorded run, not by an attack trial, on 2026-08-23 -- no attack run was fired; see redteam.why.

Each boundary above was checked by running an assertion or by reading the recorded run, not by an attack trial -- there is no untrusted field on this corpus to construct a payload against. Two of the five are red-proven, meaning the guard was removed and the failure observed rather than merely asserted while passing.

The result0 attack trials, and five boundaries checked -- two of them red-proven by removing the guard and watching all 240 snapshots fail.

The Activity Log IS the field an outside party would control in a real deployment -- field engineers, suppliers' agents and desk staff type into it -- and this kit sends it rather than hiding it, because rule 3 lives there. On this corpus it is drawn from sixteen templates by a seeded random number generator, so there is nothing adversarial in it to catch. A version pointed at real notes reopens the question and should be attacked before it ships: an instruction embedded in an engineer's note is the obvious payload, and this kit has not fired one.

HonestyWhat this does not prove

• Whether a real deployment's Activity Log -- prose an engineer or a supplier's agent writes freely -- would carry an instruction the model follows. Not applicable to the shipped corpus, and the first thing to attack if this is pointed at real notes. It is also the section rule 3 depends on, so it cannot simply be withheld.
• Whether the state file is safe under concurrency. src/state.save() replaces atomically, which is correct for one writer and is not a concurrency model; two schedulers advancing the same order have never been run and would race.
• Whether a truncated reply can ever be partially trusted. ORJ-0008-D3 returned an EMPTY body at the ceiling, so there was nothing to partially trust; src/monitor._parse rejects it outright and the run records it as a failure. That is the safe behaviour and is not the same as having measured how often a fragment would have been right.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

The carried state is written by code from the parsed task ages and the answer key's rule-3 fact. The model's status, rule and top blocker are scored and are never written back into the history the next surveillance day is judged against. And nothing in the kit acts on any of it: there is no path that reassigns a task, moves a committed date, chases a supplier or contacts a customer.

evals/run.py's per-day loop calls src/jeopardy.step() after every snapshot including one whose call FAILED. There is no code path anywhere in the kit that writes a model reply into the state store, and no writer of any kind outside results/*.json and data/state.json.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• IT IS NOT A CHECK ON WHETHER THE VERDICT IS RIGHT. The state is correct whatever the model says, which means a wrong status is reported wrongly for that day and nothing catches it -- see Eval.taxonomy's two snapshots where a narrative note was read as a change of task state.
• IT IS NOT A RATE LIMIT OR AN APPROVAL GATE. There is no apply path to gate: nothing here escalates, rebooks or contacts anybody, so there is no 'undo' because there is nothing that could 'do'.
• ⚠︎ IT IS NOT A MEASUREMENT OF ERROR PROPAGATION, AND ON THIS KIT THAT MATTERS MORE THAN USUAL. The state is advanced from the ANSWER KEY, because rule 3's fact lives only in prose. A real deployment must persist that fact itself, from its own reading, and a wrong one corrupts every later day of the order. Every lead-time figure on this page is therefore a property of each day's reading in isolation.
• It does not make the state file safe to share. One writer, one file, replaced atomically -- two schedulers on one order would race and the loser's day would vanish.
• It does not validate the answer key. The key is arithmetic, and rule 3's boundary -- 'reaches BEYOND the committed date' -- is a reading. Every arm agreed with it on all 24 commitments landing exactly ON the date, which is agreement rather than validation.

WatchedWhat is watched, and why that one

The kit measures 30 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

10 measured by the latest run 20 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

6 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

stub · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 2 chips that all say so.

ablation · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 6 chips that all say so.

triage · with the model in the path — 4 runs. Columns here are only ever compared with each other.

not a time series No two of these 4 runs measured the same system — they differ on documents, max_tokens, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 6 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-run evals/check_labels.py (free) on any change to tools/build_corpus.py, src/jeopardy.py, src/monitor.py, src/segment.py or src/select.py -- it refuses to let a run spend if the corpus, the answer key, the parser or the privacy guard has moved. Re-run all three free floors together (free) whenever the policy changes, because the headline is a DIFFERENCE and one arm re-run alone is not comparable with another's old figure. Re-run the scored eval (paid, one call per snapshot) only on a change to src/prompt.py or src/state.py.

In one lineNo framework, deliberately

No framework and no dependencies at all -- json, os, re, random, datetime, zlib, argparse, urllib, http.server and concurrent.futures from the standard library. requirements.txt names nothing and says in a comment that the emptiness is load-bearing: a line there means something under src/ or evals/ imports it. The one optional dependency is not Python -- tools/shoot_ui.mjs needs Node and puppeteer to re-take the committed screenshots.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph, and the one place it looks like one is a straight line. Each order is a chain of five steps -- one surveillance day each -- with no branching and exactly one edge between consecutive days, carrying four fields. Different orders never touch. A framework would add an orchestrator to a for-loop that already runs 48 chains wide.

The other sideWhat a framework costs you

• No scheduler. This is a monitor that carries state and is still INVOKED, not woken -- the cadence, the retry policy and the question of what to do when a surveillance day does not arrive are all outside the kit.
• No state store worth the name. data/state.json is one file replaced atomically; a framework would bring a checkpointer with a concurrency model, which this does not have.
• No routing. Every snapshot goes to the model, and this kit's own free floors say most of them need not.
• No observability beyond what evals/run.py prints and writes to results/. There is no tracing and no dashboard integration.
• No retry/backoff beyond src/adapters' own bounded retry, which covers a busy provider and a dropped connection and nothing else -- and notably NOT a reply that runs away to the ceiling, which is the failure this run actually had.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-order-jeopardy on the fast tier, with the carried state, 2026-08-23. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 3 runs on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — one period report goes whole into one call, and the prior period's carried state goes with it.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved; every order, product, task, supplier commitment, activity note and name is invented here. Every contact number is in the 07700 900000-900999 range reserved by Ofcom for fiction and never allocated to a subscriber. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineWas each slipped order surfaced before its committed date, and by how many working days

For each of the 30 orders that ultimately missed its committed date, on which surveillance day did this arm first say JEOPARDY, and how many working days was that before the date? An arm that never said it missed the order entirely.

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model. The same function all three free floors are scored through.

Every grader on these pages scored the same 240 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

For each slipped order, the first day where the arm's own answer is JEOPARDY; lead = committed working day - that day. Caught = the count with a lead of at least 1. The denominator is ORDERS, not snapshots.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl's outcome record -- which orders slipped, computed at generation time and printed on no snapshot. The lead time itself is read off each arm's OWN answers, so this grader compares an arm to the world rather than to another arm.

How tight can the band be? No tuned threshold anywhere on the model path -- the reply decides. 30 orders means the finest honest band is about 3.3 points.

Cadence: Re-run the free floors (free) on any change to tools/build_corpus.py, src/jeopardy.py, src/segment.py or src/select.py. Re-run the scored eval (paid, one call per snapshot) on any change to src/prompt.py or src/state.py.

The decisionWhen to reach for it

In one lineThe status, the rule and the top blocker, per snapshot, exact match against the computed answer key

For each of the 240 snapshots and each of the three answered fields, did the reply equal the computed answer key? Status and rule are compared exactly; the task name is compared on trimmed lower-cased text and nothing looser, because a fuzzy match would forgive an arm that invented a plausible task.

MethodHow the test was run

evals/scoring.py::score, the same pass as the lead-time grader.

Every grader on these pages scored the same 240 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 240 per field. A snapshot whose reply did not parse counts as a MISS in every field, never as an exclusion.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, computed by src/jeopardy.replay over the corpus values at generation time. This grader IS the reference, so its own error rate is not measured -- scoring it against itself would be circular.

How tight can the band be? 720 cells, so one cell is 0.139 points. Nothing here is tuned; there is no threshold to sweep.

Cadence: Whenever the lead-time grader runs -- it is the same pass.

The decisionWhen to reach for it

In one lineSnapshots whose correct status is not JEOPARDY that got one anyway

Of the 143 snapshots whose correct status is CLEAR or WATCH, how many were raised to JEOPARDY? This is the price of being early, and it is the direction that costs a coordinator their attention.

MethodHow the test was run

evals/scoring.py::score, the same pass as the reference grader.

Every grader on these pages scored the same 240 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

cells where gold status != JEOPARDY and the reply said JEOPARDY, divided by 143.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the same data/gold.jsonl, restricted to the 143 cells whose status is not JEOPARDY.

How tight can the band be? 143 rows, so one row is 0.70 points.

Cadence: Whenever the reference grader runs -- it is the same pass.

The decisionWhen to reach for it

In one lineThe same forward commitment, rewritten in words the generator never produced

When a supplier commitment is stated in a phrasing that appears nowhere in the corpus generator and that no pattern in evals/baseline.py matches, does the arm still reach the right status? This is the only grader here that separates reading the SENTENCE from matching the PATTERN.

MethodHow the test was run

evals/paraphrase.py. The free half re-scores the pattern arm for $0.00; the model half cost 24 live calls.

Every grader on these pages scored the same 240 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

status accuracy over 24 rewritten snapshots -- 12 whose commitment reaches past the committed date and 12 whose does not -- against the unchanged answer key.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the unchanged data/gold.jsonl. The rewrite states the identical fact, so no gold label moves and the two arms are scored on the same cells they were scored on before.

How tight can the band be? 24 rows means one row is 4.17 points. Nothing is tuned.

Cadence: Re-run the free half on any change to evals/baseline.py's patterns. Re-run the paid half only when the prompt changes.

The decisionWhen to reach for it