The business caseThe problem this solves

"Which trailers are still on the yard" is a SELECT and nobody needs a language model for it. The question a yard actually has to answer is the next one: which of them are now past the free time their carrier agreement grants, how many hours of that are chargeable, and -- the part that is a monitor question rather than a query -- whether the detention event has already been opened. A detention claim survives a carrier dispute on CONTEMPORANEOUS evidence: the gate log, the appointment record and the yard log as they stood when the free time ran out. Reconstructed three shifts later, it is the thing that gets thrown out. Today a clerk walks the yard report at shift change and works it out from the gate log by eye. Somebody walking the yard report at shift change: opening each open visit, finding the clock start (the later of the appointment and the gate-in, not the arrival), reading the yard log to decide which stops were the carrier's and which were ours, adding what is left to whatever was already on the clock at the last look, and remembering whether a detention event has already been raised on this visit.

The inputThe actual yard snapshots

The corpus is 120 yard snapshots, 0.44 MB (json 1 · jsonl 1 · txt 120). A YARD'S DETENTION FILE IS THE MOST COMMERCIALLY SENSITIVE ARTEFACT AN OPERATION KEEPS. It is the evidence a carrier bills against and the evidence a shipper disputes with, it names drivers, and there is no public one -- for the same reason there is no public corpus of anyone's freight invoices. Generating it also bought the one thing a captured corpus cannot give: the answer key is src/detention.step's output over the planted intervals, so an interval rule this fiddly cannot carry its author's misreading into the score. ⚠︎ AND THE RULES ARE INVENTED. The eight rules, the free-time hours and the per-hour rates reproduce no carrier agreement, no tariff, no broker contract and no regulator's guidance, and name none. The catalogue row this kit was built from (logi:YDA-003) records the free-time value as an OPERATOR-SUPPLIED clock and explicitly BLOCKS a silent default; that is implemented as Rule D-6 and measured as context_incomplete_recall_pct rather than asserted. The rule text is reproduced in full on all 120 pages precisely so it can be read, disbelieved and replaced.

Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated yard snapshot for an AI use-case
kit; it reproduces no carrier agreement, no tariff and no real facility. The detention
terms are ILLUSTRATIVE.

Visit
----------------------------------------------------------------
Facility                       : DC-ONT-03 (Inland Empire DC)
Visit reference                : VIS-0001
Equipment                      : 53' reefer, unit 510944
Carrier (SCAC)                 : KNLX
Move type                      : LIVE LOAD
Gate in (arrival)              : 2026-03-04 05:30
Appointment                    : 2026-03-04 08:45
Clock start                    : 2026-03-04 08:45  (later of appointment and gate-in)
Gate out (departure)           : -- still on site
Door                           : 136

Detention Terms
----------------------------------------------------------------
Carrier agreement              : MSA-KNLX-2025
Free time on file              : 3.00 hours from clock start
Detention rate on file         : 85.00 USD per hour, 15-minute increments

Rule D-1  Free time runs from the CLOCK START, which is the later of the appointment time and the
          gate-in time. A carrier that arrives early does not start its own clock early.
Rule D-2  Detention accrues once free time is exhausted while the equipment is still on site, in
          15-minute increments, at the rate on file.
Rule D-3  The clock STOPS for any interval the yard event log attributes to the CARRIER: the driver
          leaving the site, no power unit present, an hours-of-service break, or the carrier
          declining to work the door it was given. Those hours do not count.

The outcomeWhat a good result looks like

Every visit past its free time is on the watchlist with the hours already accrued, and exactly one detention event exists per visit -- opened on the run that first saw the breach, not re-opened on every run after it.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• You already know which trailers are over their free time and only need the list — a SQL query, or the b000 floor in this repo
  Elapsed against a threshold is arithmetic. On this corpus b000 gets 80.83 pct of statuses right for nothing at all.
• Your yard log is structured -- every hold carries a coded reason from a closed list — the b002 floor: keyword-free, code-only attribution on your own codes
  The model's entire margin on this corpus is reading free prose. Take the prose away and there is nothing left to buy: the floor already ties on status and the gap on hours is attribution.
• Your yard log is prose an inbound clerk types, and it mentions the driver whoever's fault the hold is — this kit, with the carried state
  That is exactly the corpus these figures were measured on, and it is where the clock and hours gap comes from: 99.17 pct against 85.00 pct on exact hours, and 99.34 pct against 94.06 pct of the yard's detention value.
• You want the watch to actually wake up on a schedule — your own scheduler -- cron, Airflow, Temporal, whatever already runs
  evals/run.py is INVOKED, not woken. Nothing in this kit detects a missed run, back-fills it, or marks its readings late.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-23. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-23 — r001-detention-watch. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with NO key configured reproduces: the 120 snapshots and the answer key (regenerated from the seed in under a tenth of a second), all nine pre-flight assertions, all three free floors scored end to end, the wiring stub, and the local UI at 127.0.0.1:9009 including what run r001 recorded for every reading. Clone to first scored floor result measured at 0.39 seconds of wall clock on the author's machine. What it CANNOT reproduce without a key is a model column of its own -- and on 2026-08-23 neither could the author, because the shared provider account ran out of balance.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per reading: one HTTP request carrying the assembled prompt, and the reply parsed to a status, a clock state, an hours figure and a raise/hold call. Splitting the snapshot into sections, dropping the one section no field asks for, parsing the times and advancing the clock all happen outside this measurement and cost no network at all. THE TAIL IS THE STORY: p95 (25.5s) is 4.7 times p50 (5.4s). Reply length here is driven by how hard the INTERVAL ARITHMETIC is, not by how long the page is -- every snapshot is within 6 pct of every other in size (3775 to 3989 bytes) and replies ran to 7,760 output tokens at the top.

Current processWhat it replaces

Somebody walking the yard report at shift change: opening each open visit, finding the clock start (the later of the appointment and the gate-in, not the arrival), reading the yard log to decide which stops were the carrier's and which were ours, adding what is left to whatever was already on the clock at the last look, and remembering whether a detention event has already been raised on this visit.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• 3 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one visits (3 scheduled runs each) is

No split, and no chunking. The unit is a VISIT -- three consecutive scheduled runs processed strictly in order, because run 3's prompt contains a clock produced by run 2. Each snapshot goes whole into one call, minus the one section no field asks for.

SetupWhat the setup figure measured

There is no index and no retrieval step -- the population is re-read whole on each scheduled run. tools/build_corpus.py writes 120 documents and the answer key from a fixed seed in under a tenth of a second with no clock read and no model called; nothing is embedded, ranked or cached.

LicenceLicence

MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved; every trailer, carrier code, agreement reference, driver, free-time figure, rate and yard-log line is invented here. Verified against the repository's own LICENSE file on 2026-08-23.

Bring your ownBring your own yard snapshots

Point tools/build_corpus.py at your own yard, or replace data/corpus/*.txt and data/gold.jsonl directly. Keep the seven section headings in src/segment.py::SECTIONS -- the parser asserts all seven in every document before a run may spend -- and keep the Detention Terms block, because the rule the model applies is READ OFF THE PAGE rather than baked into the prompt, which is what lets you swap your own agreement in without touching src/prompt.py. gold.jsonl needs one row per snapshot carrying hours_counted, carrier_suspended_hours, free_time_hours, the three booleans and the four answers; evals/check_labels.py re-derives the answers from src/detention.step and refuses if they disagree, so a hand-written key is caught rather than trusted.

⚠︎ And what stops being true when you do: The measured figures on this page do not travel with your corpus, and two of them are corpus properties rather than model properties. EVERY figure here is per a SIX-HOUR cadence: the WATCH band is defined as the cadence itself, so a yard on a shift cadence (8h) or an hourly one is scoring a different question with the same words. And the population here does not change between runs, which is not what a real yard does.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 7,068 characters assembled, 3,092 are instructions — 44% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

TWO MESSAGES, JOINED IN SEND ORDER. The provider receives a system message and a user message; prompt_verbatim prints both, separated by a blank line, because publishing only the second would be publishing most of a prompt. Replayed from the kit's own src/prompt.build() for VIS-0006-R2 with the carried state that reading was actually given, rather than logged by the run. What makes the replay checkable is that the section list it produces is the list the run recorded in sections_used -- Synthetic Record, Visit, Detention Terms, Dwell Position, Yard Event Log, Operational Notes -- and that src/prompt.build() is the only thing in the kit that assembles a prompt, so there is no second path a run could have taken. ⚠︎ WHAT IT IS *NOT* CHECKED AGAINST, SAID PLAINLY: the run's own prompt_parts records ONE size, 6883 characters, and it belongs to whichever reading finished first under 12 concurrent workers -- not to VIS-0006-R2, whose assembled user message is 7070 characters. The harness stores one exemplar, not one per reading, so no per-reading size comparison is available and none is claimed.

Full promptThe prompt in full

You are a scheduled yard detention watch. You apply a written detention rule to one trailer visit at one scheduled run. You answer with one JSON object and no other text.

You are the scheduled detention watch for one distribution centre. It wakes every six hours, re-reads
every trailer still open on the yard, and reports each one. You are reading ONE trailer visit at ONE
scheduled run.

The detention rules, the free time on file and the rate on file are reproduced in the snapshot below.
Apply them exactly as written. The clock is a RUNNING TOTAL that survives between scheduled runs, and
you cannot see the earlier runs; what is known about them is stated under "Carried state" and is the
only history available to you. Do not assume anything about earlier runs beyond it.

How to count:
 - "Hours counted in this window" is the elapsed clock-eligible time since the previous scheduled
   run. It is already computed for you; do not re-derive it from the gate times.
 - From it, SUBTRACT every hour the Yard Event Log in this window attributes to the CARRIER
   (Rule D-3). Do not subtract hours attributable to the FACILITY (Rule D-4). An interval runs from
   the entry that opens it to the entry that closes it. The log shows only the entries raised in
   THIS window: a closing entry with no opening entry belongs to a stop that was already running,
   and the carried state says whether that stop was carrier-attributable and therefore whether the
   hours before it closed are excluded.
 - ADD what is left to the hours the carried state says were already on the clock.
 - Detention is whatever that total exceeds the free time on file, in quarter hours. If it does not
   exceed it, detention is 0.
 - If no free time is on file, or gate-in evidence is missing, the visit is CONTEXT_INCOMPLETE:
   status CONTEXT_INCOMPLETE, clock NOT_STARTED, detention_hours 0, raise_event NO. Never assume a
   default free-time figure (Rule D-6).

Answer with a single JSON object and nothing else:

{"status": "CLEAR|WATCH|DETENTION_OPEN|SUSPENDED|CONTEXT_INCOMPLETE",
 "clock": "RUNNING|STOPPED|NOT_STARTED",
 "detention_hours": <hours past free time now on the clock, a multiple of 0.25>,
 "raise_event": "YES|NO",
 "rationale": "one sentence, naming the rule you applied and the hours you counted"}

Precedence for "status", applied in this order: CONTEXT_INCOMPLETE beats everything; then
DETENTION_OPEN if any detention has accrued, even if the clock is currently stopped; then SUSPENDED
if a carrier-attributable stop is open at the snapshot and the equipment is still on site; then
WATCH if less free time remains than the six-hour cadence (Rule D-8); otherwise CLEAR.

"clock" is NOT_STARTED before the clock start is reached or when the context is incomplete, STOPPED
once the equipment has gated out or while a carrier-attributable stop is open at the snapshot, and
RUNNING otherwise.

"raise_event" is YES only on the run that OPENS the detention event -- detention has accrued and the
carried state does not already say an event was opened. On every later run it is NO (Rule D-5).

Carried state
----------------------------------------------------------------
As at the previous scheduled run, 0.00 hours were already on this visit's detention clock, carrier-attributable stops already deducted. No carrier-attributable stop was open when that run ended. No detention event has been opened for this visit yet. It was reported CLEAR.

Yard snapshot
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated yard snapshot for an AI use-case
kit; it reproduces no carrier agreement, no tariff and no real facility. The detention
terms are ILLUSTRATIVE.

Visit
----------------------------------------------------------------
Facility                       : DC-ONT-03 (Inland Empire DC)
Visit reference                : VIS-0006
Equipment                      : 53' reefer, unit 642835
Carrier (SCAC)                 : JBHT
Move type                      : LIVE UNLOAD
Gate in (arrival)              : 2026-03-04 05:15
Appointment                    : 2026-03-04 06:15
Clock start                    : 2026-03-04 06:15  (later of appointment and gate-in)
Gate out (departure)           : -- still on site
Door                           : 138

Detention Terms
----------------------------------------------------------------
Carrier agreement              : MSA-JBHT-2025
Free time on file              : 4.00 hours from clock start
Detention rate on file         : 85.00 USD per hour, 15-minute increments

Rule D-1  Free time runs from the CLOCK START, which is the later of the appointment time and the
          gate-in time. A carrier that arrives early does not start its own clock early.
Rule D-2  Detention accrues once free time is exhausted while the equipment is still on site, in
          15-minute increments, at the rate on file.
Rule D-3  The clock STOPS for any interval the yard event log attributes to the CARRIER: the driver
          leaving the site, no power unit present, an hours-of-service break, or the carrier
          declining to work the door it was given. Those hours do not count.
Rule D-4  The clock does NOT stop for an interval attributable to the FACILITY: a dock fault,
          missing labour, a systems outage, congestion, or no door available. Those hours count in
          full -- that is what detention is for.
Rule D-5  A detention event is logged ONCE per visit, on the first scheduled run at which free time
          is exhausted. A later run reports the event already open; it does not open a second one.
Rule D-6  Free time is an OPERATOR-SUPPLIED value read from the carrier agreement. Where no free
          time is on file, or where gate-in evidence is missing, the visit is CONTEXT_INCOMPLETE:
          no detention accrues, no event is logged, and no default free-time figure may be assumed.
Rule D-7  Once the equipment gates out, the clock stops at the gate-out time.
Rule D-8  A visit is on WATCH while less free time remains than the watch cadence -- that is, when
          it will be exhausted before the next scheduled run if the clock keeps running. On the
          six-hourly cadence this kit ships on, that means under 6.00 hours of free time left.

Dwell Position
----------------------------------------------------------------
Snapshot taken                 : 2026-03-04 12:00   (scheduled run 2 of this visit)
Watch cadence                  : every 6 hours (06:00 / 12:00 / 18:00 local)
Previous scheduled run         : 2026-03-04 06:00
Hours counted in this window   : 5.75
Elapsed on the clock at this snapshot, before any exclusion : 5.75 hours

Yard Event Log
----------------------------------------------------------------
Entries raised in this window only.

08:45  dispatch here put two appointments on the same door, this one is held
11:00  door cleared, this trailer worked
11:15  driver left the site for a federally required 30-minute break

Operational Notes
----------------------------------------------------------------
Carrier has asked to be told before any detention event is opened on this visit.

Raw responseThe raw response

{
 "status": "DETENTION_OPEN",
 "clock": "STOPPED",
 "detention_hours": 1.0,
 "raise_event": "YES",
 "rationale": "5.75 window hours minus 0.75 carrier-attributable break (Rule D-3) leaves 5.00 clock hours, exceeding 4.00 free time by 1.00 hour, so detention opened this run (Rule D-5) while the clock is stopped by the open carrier stop."
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch a yard trailer's free time before it runs out — 120 yard snapshots. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Every cell is compared for exact equality against a gold set computed by tools/build_corpus.py from the visit model -- the intervals, their attribution and the gate times -- and re-derived from src/detention.step by evals/check_labels.py before any run may spend. Detention hours are compared on the quarter-hour billing increment, so the comparison is exact rather than nearly exact. No model grades anything, here or anywhere in this kit.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran all 247 calls is kept out of these tables per this estate's naming rule, so nothing here is what was paid. The real spend for this kit's whole build is on the shared call ledger, not on this page.

Grading unitWhat the grading figure prices

The GRADER costs nothing: evals/scoring.py makes no model call, and neither do the three free floors it is compared against nor the pre-flight. The cost of the RUN is a different figure and lives in lens 07 -- pricing the ruler in the units of the thing it measures is the trap this field exists to name.

The gradersThree ways to grade

⚑ THE MODEL TIES THE STRONGEST FREE FLOOR ON THE HEADLINE BAND AND WINS EVERYTHING UNDERNEATH IT. Status: 95.00 pct against 95.00 pct -- 114 of 120 each, a dead tie. Clock: 99.17 against 86.67. Detention hours, exact: 99.17 against 85.00. Raise/hold: 100.00 against 98.33. Money: the yard's accrued detention is $22,687.50 and the model reports $22,837.50 (0.66 pct out) against the floor's $24,035.00 (5.94 pct out). ⚑ AND THREE FLOORS RATHER THAN ONE, BECAUSE ONE CANNOT SEPARATE THE MEMORY FROM THE MODEL. b000 is the ageing rule a yard already runs -- elapsed against free time, no memory, and a DEFAULT free-time figure where none is on file: 80.83 pct status, 70.00 pct raise/hold, and 36 duplicate raises in 92 quiet readings, because with no memory of an open event it re-opens one on every run. b001 is that same rule GIVEN the carried state: duplicates fall from 36 to 3 and raise/hold goes 70.00 -> 97.50. That jump is what MEMORY ALONE buys, with the model held out of it, and it is the largest single movement anywhere on this page. b002 adds reading the yard log. ⚠︎ BOTH FLOORS THAT ABSTAIN SCORE 100 PCT ON CONTEXT-INCOMPLETE AND BOTH THAT GUESS SCORE 0. b000 and b001 fill a missing free-time value with an assumed 2.00 hours -- exactly what a threshold report configured once and forgotten does -- and get all 18 of the readings with nothing on file wrong, accruing detention against a number nobody supplied. That is not a model finding; it is the row's own guardrail, measured.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same reading (one visit, at one scheduled run), the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

247 live calls were attempted for this kit and 244 returned something: 6 calibration (c000, at an 8,000-token ceiling on the two hardest visit chains), 120 scored, 120 the stateless control (117 answered), and 1 for a UI screenshot that the provider REFUSED. NOTHING WAS DISCARDED BY CHOICE and no run was re-fired -- the corpus defect this kit found (see Data.breaks_on) was caught by reading generated documents BEFORE any call was made, which is the whole argument for reading your own corpus first. The 2 discarded calls are the provider's own refusals: a 402 Insufficient Balance and a 429, both of which return no completion and are billed for none. Everything else -- all three free floors, the wiring stub, the scorer, the pre-flight -- is pure code and costs $0.00. Figures are projected onto the Google Gemini 3 Flash card, not the real spend.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, and most of them are reasoning. 89.9 pct of this run's output (103,557 of 115,144) was provider-side reasoning left at the default, and output is 77.1 pct of the projected bill on the shared card. What you are paying for is the model doing interval arithmetic.
• THE CADENCE, which multiplies everything else. One reading is one call; the watch wakes 4 times a day; the bill is rows x runs, not rows.
• THE OPEN POPULATION, not the yard's size. A visit that has gated out and been validated leaves the watch and stops costing anything. A yard that never closes its events pays for them forever.
• The snapshot itself, barely. Input averages 1708 tokens and every document in this corpus is within 6 pct of every other in size.

Your volumeWhat it costs at your volume

LINEAR IN ROWS x RUNS, AND THAT IS THE WHOLE WARNING. Ten times the open visits is ten times the calls at the same cadence -- there is no batching, no cache and no early exit, because every visit is re-read whole on every run by design. A yard holding 400 open visits on this six-hourly watch is 1,600 calls a day, about $5.97 a day on the shared projection card. Nothing about the per-call price changes; the multiplier is the schedule, which is why this kit's environment ladder makes cadence a declared decision rather than a deployment detail.

Model choiceWhy this model

It is the tier the shared connection was already pointed at, and the kit's whole claim is that swapping it is one line of .env and one more run. No comparison across tiers was attempted: the shared provider account hit Insufficient Balance immediately after the control arm, so a second tier was not run and nothing on this page claims one.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every stateful number on these pages: 120 readings, one completion call each, one tier. The 120-call stateless control, the 6 calibration calls and the 1 refused screenshot call are accounted for separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

12 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pythe corpus generator — a swap seam

Generates 40 trailer visits x 3 scheduled runs = 120 snapshots from a fixed seed (SEED = 20260823). A third of the yard-log intervals are built to STRADDLE a scheduled run, so the snapshot carrying the 'resumed' line does not carry the 'halted' line that explains it -- that is the experiment. The gold labels are src/detention.step's output over the planted inputs, never typed.

You change it to: The visits, the phrasing pools and the seed. Keep the seven section headings or src/segment.py's assertion refuses to start.

# Generate the corpus and the answer key. Deterministic: one seed, one corpus, forever.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
SEED = 20260823
VISITS = 40
DAY = "2026-03-04"
RUN_HOURS = (6.0, 12.0, 18.0)
RULE = "-" * 64

src/detention.pythe detention arithmetic — a swap seam

The rule as pure code: hours counted this window, minus the carrier-attributable stops, added to whatever was already on the clock, against the free time on file. No model, no judgement. ⚠︎ The rules, the free-time hours and the rates in it are INVENTED and the kit ships that as a stated limitation -- see Data.why_this_corpus.

You change it to: RULE_TEXT and the precedence in step(). The rule text is reproduced in full on all 120 pages precisely so it can be read, disbelieved and replaced with the agreement you actually signed.

# The detention rule as arithmetic. Pure code, no model, no dates that need a library.

CLEAR = "CLEAR"
WATCH = "WATCH"
DETENTION_OPEN = "DETENTION_OPEN"
SUSPENDED = "SUSPENDED"
CONTEXT_INCOMPLETE = "CONTEXT_INCOMPLETE"
STATUSES = (CLEAR, WATCH, DETENTION_OPEN, SUSPENDED, CONTEXT_INCOMPLETE)
RUNNING = "RUNNING"
STOPPED = "STOPPED"
NOT_STARTED = "NOT_STARTED"

src/state.pythe carried state — a swap seam

SEAM 2 -- the thing that makes this a monitor. Four scalars per visit (hours on the clock, whether a carrier stop is open, whether an event has been opened, the status last reported), written from the arithmetic and never from the model's reply, and rendered into one English sentence for the prompt. It costs the same on run 400 as on run 2, which matters here because the watch wakes four times a day.

You change it to: What is carried between scheduled runs, and how it is worded to the model. Four scalars today. ⚠︎ THE WORDING IS LOAD-BEARING AND THIS RUN PAID FOR IT: 'hours were already on this visit's detention clock' was read once as hours of DETENTION, and that is the run's only genuine reasoning error.

# The carried state -- the thing that makes this a monitor and not another one-shot classifier.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEFAULT_PATH = os.path.join(HERE, "data", "state.json")
def load(path=DEFAULT_PATH):
def save(store, path=DEFAULT_PATH):
def for_visit(store, visit_id):
def describe(state):

src/segment.pythe section splitter

Splits a snapshot into its seven named sections. Pure code; evals/check_labels.py asserts all seven are present in all 120 documents before a run may spend.

# Split a yard snapshot into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "Visit", "Detention Terms", "Dwell Position",
RULE = "-" * 64
def split(text):
def render(secs):

src/select.pythe section selector

Decides which sections reach the model. Driver Contact -- the driver's name, mobile and CDL number -- is mapped by no field and is subtracted unconditionally by _fallback(), so the one section whose subject is a PERSON never leaves the machine even when a source-system rename makes every hint match nothing.

# Pick which sections of a snapshot are sent. Pure code -- the last deterministic step before the

BANNER = "Synthetic Record"
VISIT = "Visit"
TERMS = "Detention Terms"
POSITION = "Dwell Position"
EVENTS = "Yard Event Log"
DRIVER = "Driver Contact"
NOTES = "Operational Notes"
NEVER_SENT = (DRIVER,)
SECTION_HINTS = {

src/prompt.pythe prompt

Three parts: the fixed instruction, the carried-state sentence, and the selected sections in document order. The stateless build replaces the sentence with a line saying no history is available -- byte-identical everywhere else, which is what makes the control a control, and evals/check_labels.py asserts exactly that.

# Assemble the one prompt this kit sends. No template engine, no framework -- string concatenation

INSTRUCTION = """\
def build(text, carried, stateless=False):

src/adapters/__init__.pythe model adapter — a swap seam

SEAM 1 -- the model. Raw HTTP for every provider, standard library only. Returns input/output tokens, reasoning tokens and finish_reason beside the text, and separates a transport failure from an HTTP status so a dropped connection is retried rather than recorded as a lost reading.

You change it to: PROVIDER, BASE_URL, API_KEY and MODEL in .env. Adding a provider is one function and one entry in PROVIDERS; it must return token counts, because lens 05 publishes them and lens 07 prices them.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/watch.pythe watch

One visit, one scheduled run, one call. Parses the clock times, the counted hours and the free time off the page with a regex (the model is never asked to read a timestamp), assembles the prompt, calls the model, parses the reply. Holds MAX_TOKENS = 24000, the published ceiling.

# One visit, one scheduled run, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = ("You are a scheduled yard detention watch. You apply a written detention rule to one "
MAX_TOKENS = 24000
FIELDS = ("status", "clock", "detention_hours", "raise_event")
def documents():
def visits():
def load_doc(doc_id):
def position_of(text):

src/app.pythe local UI

One visit, one scheduled run, its carried state and its verdict, on 127.0.0.1:9009. Renders with no key. It shows the carried sentence verbatim, the free floor's answer beside the model's, and -- added the day the shared provider account ran out of balance -- a second button that replays what run r001 actually answered, straight off the committed result file, labelled as a replay.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
PORT = int(os.environ.get("PORT", "9009"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-detention-watch")
GOLD_ROWS = {r["doc_id"]: r for r in
def carried_for(doc_id):
class H(BaseHTTPRequestHandler):
def main():

evals/baseline.pythe three free floors — a swap seam

b000 the ageing rule a yard already runs, with a DEFAULT free time where none is on file; b001 the same rule given the carried state, which isolates what memory alone buys; b002 the strongest free version -- carried state, the log parsed into intervals, attribution by keyword table, and an abstention where nothing is on file. 0 calls, $0.00, all three scored through the identical scorer.

You change it to: CARRIER_WORDS and OPEN_MARKERS -- the keyword table the model is measured against. Widen it and the floor gets stronger, which is the direction this kit wants: a floor built to lose measures nothing.

# THE FREE FLOORS. Three of them, none a strawman, all of them $0.00.

MODES = ("ageonly", "ageonly-mem", "keyword-mem")
ASSUMED_FREE_TIME_HOURS = 2.0
CARRIER_WORDS = ("driver", "tractor", "bobtail", "power unit", "dispatch",
OPEN_MARKERS = ("halted", "held", "hold", "cannot", "will not", "no door", "no power",
def _num(pat, text, cast=float):
def _log_lines(text):
def _is_carrier(line):
def _is_close(line):
def _carrier_suspended_hours(text, carried, window_start_h, snapshot_h):

evals/scoring.pythe scorer

Exact match per cell against the computed gold, split ways an average would hide: the four fields, the two raise directions counted apart, the memory-dependent subset, the context-incomplete recall, and the yard's detention exposure in dollars. No judge model.

# Score a run against the answer key. Pure code, exact match per cell. No model grades anything.

FIELDS = ("status", "clock", "detention_hours", "raise_event")
def _pct(n, d):
def _hours(v):
def score(records, golds):

evals/check_labels.pythe pre-flight

Everything that must be true before a run may spend: the seven sections parse, every visit's run sequence is complete and gap-free, no snapshot restates its own carried state (the experimental control), the privacy guard holds AND would fail without it, the two prompt builders differ on exactly one line, no code path approves or pays a charge, and the answer key replays from src/detention.step.

# Everything that must be true BEFORE a run is allowed to spend. Free, and it runs in seconds.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
FAILED = []
def check(name, ok, detail=""):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 1708 input and 959 output tokens per reading (one visit, at one scheduled run), all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

No untrusted external input surface in the shipped version. Every field that reaches the prompt comes from tools/build_corpus.py's fixed seed, including the Operational Notes, which are one of three fixed sentences. src/select.py names the notes as the field an outside party WOULD influence in a real deployment -- prose an inbound clerk types into the yard system -- and sends them deliberately, so the surface is visible rather than hidden. On this corpus nobody outside the repository wrote a byte of it.

The provider key lives only in .env (gitignored) or the shared repo-root .env, read by src/config.py and never logged. config.sources() reports WHICH file a value came from and never the value. config.save() opens the file 0600 before the write rather than chmod-ing it after, so the key never exists at the default umask even for an instant. src/app.py redacts the key and the base URL out of any provider error before it reaches the browser -- which is not a theoretical claim here: the provider-refused screenshot on this page IS that path, captured live.

The experimentWe did NOT attack it — and this is the surface we left open

An indirect prompt injection needs a field somebody outside can write into, and this kit has exactly one: the Operational Notes an inbound clerk types into the yard system. It is SENT rather than hidden, because hiding a surface does not close it. No attack was fired at this kit and none is claimed. The three boundaries below were checked in code on 2026-08-23, and only the first of them is red-proven in both directions rather than argued from an absent code path.

The first row is the only one measured in both directions. The other two are guarantees about what is ABSENT — no code path approves, releases or pays a charge, and no code path feeds a model reply back into the detention clock — and absence is checked by asserting names it knows, which is weaker than a run and is written here as such.

The result0 attack trials, three boundaries checked -- and the privacy boundary red-proven by removing the guard and watching all 120 snapshots leak a commercial driver's name, mobile and CDL number.

The Operational Notes ARE the field an outside party would influence in a real deployment, and this kit sends them rather than hiding them -- one of the three shipped sentences is even instruction-shaped ("Carrier has asked to be told before any detention event is opened on this visit"), chosen for exactly that reason. But on this corpus they are one of three sentences from a seeded generator, so there is nothing adversarial in them to catch. A version pointed at real clerk prose reopens the question and should be attacked before it ships. What IS measured is the privacy boundary, in both directions, before every run.

HonestyWhat this does not prove

• Whether a real deployment's Operational Notes -- prose an inbound clerk writes freely -- would carry an instruction the model follows. Not applicable to the shipped corpus, and the first thing to attack if this is pointed at a real yard. The shipped note asking to be warned before an event is opened is the shape to probe.
• Whether the state file is safe under concurrency. src/state.save() replaces atomically, which is correct for one writer and is not a concurrency model; two watches on one yard have never been run and would race. On a RUNNING CLOCK a lost run is a permanently wrong number of billable hours rather than a missing row.
• Whether a truncated reply can ever be partially trusted. The stateless control lost one call to the ceiling and src/watch._parse rejected it outright, which is the safe behaviour and is not the same as having measured how often a fragment would have been right.
• Whether the grep-shaped no-payment assertion would catch an approval path written under a name it does not know. It asserts the absence of seven names; a path called something else would pass it.
• What the provider retains. The prompt carries no personal data by construction -- the Driver Contact section never leaves -- but what a vendor keeps of a request is outside this repository and nobody here has verified it.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

No carrier-payment release, non-configurable. This kit produces a watchlist status, a clock state, an accrued-hours figure and a raise/hold call for a yard manager to read. It never approves, releases, adjusts, suppresses or pays a detention charge, never negotiates with a carrier and never writes to a TMS, and there is no setting that makes it. Separately, and just as non-configurable: the free-time value is OPERATOR-SUPPLIED. A visit whose agreement carries no detention clause is reported CONTEXT_INCOMPLETE and accrues nothing -- there is no default figure in this kit.

Everywhere and nowhere -- it is a property of what is ABSENT. The only writers in the kit are evals/run.py (results/*.json) and src/state.save (data/state.json). src/detention.step() is the only thing that touches the clock, and it is called by evals/run.py after every reading including one whose call FAILED.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• IT IS NOT A CHECK ON WHETHER THE ANSWER IS RIGHT. The clock is correct whatever the model says, which means a wrong reading is a wrong watchlist row and not a corrupted history. Those are different problems and only the first one is on this page.
• IT IS NOT A SCHEDULER. evals/run.py is invoked; it is not woken. Nothing here detects a missed run, back-fills it or marks its readings late -- which is the half of a monitor this variant exists to name, and this kit has the same hole.
• IT IS NOT AN AUTHORITY ON DETENTION. The eight rules, the free-time hours and the rates are invented for this kit and reproduce no carrier agreement, tariff or regulator's guidance.

WatchedWhat is watched, and why that one

The kit measures 34 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

16 measured by the latest run 18 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (cadence_hours 6, context_incomplete_cells 18, documents 120, memory_cells 61, quiet_cells 92, raise_cells 28, readings_scored 120, stateless False, visits 40) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

7 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

triage · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 3 runs measured the same system — they differ on floor, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

triage · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on context_incomplete_cells, documents, max_tokens, memory_cells, quiet_cells, raise_cells, readings_scored, stateless, visits — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 16 chips that all say so.

DeviationsWhat deviated

0 breaches across 7 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-run evals/check_labels.py (free, under a second) on any change to tools/build_corpus.py, src/detention.py, src/segment.py or src/select.py -- it refuses to let a run spend if the corpus, the answer key, the experimental control, the privacy guard or the no-payment property has moved. Re-run the scored eval AND its stateless control together (paid, 240 calls) on any change to src/prompt.py or src/state.py: the headline is a difference, so one arm re-run alone is not comparable with the other's old figure. All three free floors cost nothing and should be re-run on every change, in both directions -- a floor that gets stronger is a finding, not a regression.

In one lineNo framework, deliberately

No framework and no dependencies at all -- json, os, re, random, datetime, argparse, urllib, http.server and concurrent.futures from the standard library. requirements.txt names nothing and says in a comment that the emptiness is load-bearing: a line there means something under src/ or evals/ imports it.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph, and the one place it looks like one is a straight line. Each visit is a chain of three readings -- 06:00, 12:00, 18:00 -- with no branching and exactly one edge between consecutive runs, carrying four scalars. Different visits never touch. A framework would add an orchestrator to a for-loop that already runs 40 chains wide.

The other sideWhat a framework costs you

• No scheduler. This is a watch that carries state and is still INVOKED, not woken -- and the cadence is the single most important thing about it. That is the honest cost of staying a folder of readable Python, and it is stated on the kit's own page rather than implied by silence.
• No missed-run detection, no back-fill, no late marking. All three are what a scheduling engine gives you and none of them exists here.
• No concurrency model for the state store. data/state.json is one file replaced atomically -- correct for one writer, and a yard with two watches running is two writers.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-detention-watch on the same tier, memory removed (THE CONTROL), 2026-08-23. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 5 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

3 runs not plotted. b000-detention-watch-ageonly, b001-detention-watch-ageonlymem, b002-detention-watch-keywordmem recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the population is re-read whole on each scheduled run, and the prior run's carried state is what makes a reading a CHANGE.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved; every trailer, carrier code, agreement reference, driver, free-time figure, rate and yard-log line is invented here. Verified against the repository's own LICENSE file on 2026-08-23. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe status, the clock, the accrued hours and the raise/hold call, per reading, exact match against the computed answer key

For each of the 120 readings and each of the four answered fields, did the reply equal the computed answer key? Status, clock and the raise/hold call are compared exactly; the accrued hours are compared on the quarter-hour billing increment, and a reply that is not a multiple of a quarter hour is a MISS rather than a zero -- 0.00 is a meaningful answer here (it is what every CLEAR and every CONTEXT_INCOMPLETE reading gets) and a parse failure must not be scored as one.

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model. The same function all three free floors are scored through.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 120 per field. A reading whose reply did not parse counts as a MISS in every field, never as an exclusion.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, computed by tools/build_corpus.py over the planted intervals at generation time and re-derived from src/detention.step by evals/check_labels.py before any run may spend. This grader IS the reference, so its own TPR and TNR are not measured -- scoring it against itself would be circular.

How tight can the band be? No tuned threshold anywhere on the model path -- the reply decides. The denominators are printed beside every rate because two of them are small: 28 readings that must open an event and 18 with nothing on file, so one row moves the first by 3.6 points and the second by 5.6.

Cadence: Re-run evals/check_labels.py (free) on any change to tools/build_corpus.py, src/detention.py, src/segment.py or src/select.py. Re-run the scored eval AND its stateless control together (paid, 240 calls) on any change to src/prompt.py or src/state.py -- the headline is a difference, so one arm re-run alone is not comparable with the other's old figure. All three floors are free and should be re-run on any change at all.

The decisionWhen to reach for it

In one lineThe two raise directions, counted apart

Two counts over two different denominators. A MISSED raise: of the 28 readings whose correct answer is to open the detention event, how many did not. A DUPLICATE raise: of the 92 readings that must not open one, how many did.

MethodHow the test was run

evals/scoring.py::score, in the same pass as the exact-match grader.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

missed_raise_pct = missed / 28; duplicate_raise_rate_pct = duplicates / 92. Never averaged and never combined into an F-score.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl's raise_event column, computed by src/detention.step. This grader is scored against the reference and is not itself the reference.

How tight can the band be? 28 readings must open an event and 92 must not, so the finest band either rate can support is 3.6 and 1.1 points respectively. A 'zero missed raises' claim on this corpus is a claim about 28 readings and nothing wider.

Cadence: Free. Re-run with every scored run, and re-run the floors whenever src/state.py changes -- the duplicate count is the most memory-sensitive figure on this page.

The decisionWhen to reach for it

In one lineThe readings whose answer is not on their own page

The same exact-match comparison, restricted to the 61 readings the generator marked memory_dependent: hours already on the clock that this snapshot does not restate, a carrier stop that opened before this window, or an event already opened. It answers one question -- is the carried state doing anything, or is the corpus easy?

MethodHow the test was run

evals/scoring.py::score, same pass, grouped by the flag tools/build_corpus.py set at generation time.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 61 per field, over the memory_dependent subset only.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the same data/gold.jsonl, sliced by the memory_dependent flag. Scored against the reference; not itself the reference.

How tight can the band be? 61 cells, so the finest band this subset supports is 1.6 points. Two of the three arms on it are within one cell of each other on status, which is why the raise column is the one to read.

Cadence: Free, every run. Re-derive the flag whenever tools/build_corpus.py changes -- it is a property of the corpus, and a corpus with no memory-dependent readings would make this whole kit unfalsifiable (evals/check_labels.py asserts there are at least 40).

The decisionWhen to reach for it