The business caseThe problem this solves

A programme office owes a government customer a list of data items, each with a date. The list itself is not hard to read: a spreadsheet with conditional formatting will tell you what is due in the next fortnight. What it will not tell you is what has CHANGED since last Monday — whether the date under a line moved, whether this is the first time a line has been raised or the fourth with nothing having shifted, or whether the 'revised due date' somebody typed in is carried by a contract modification that actually exists. Those are the questions the pass is really for, and none of them is answerable from the row in front of you. The Monday morning pass down a contract deliverable register — thirty open CDRL lines, each with a due date, a submission status and a paragraph of somebody's notes — deciding which ones to put in front of the programme analyst this week and which to leave standing.

The inputThe actual deliverable records

The corpus is 120 deliverable records, 0.52 MB (json 2 · jsonl 1 · txt 120). A deliverable register is the cleanest case of a population that is re-read whole: every open line is present on every run, nothing arrives as an event, and the finding is never in a row — it is in the difference between this week's row and last week's. It also has a genuine off-page fact. A 'revised due date' prints identically whether or not a contract modification carrying it has been executed, because the field is filled in by whoever asked for the relief and the modification is a different document in a different system. That single boolean decides which date the deliverable is judged against, and it lives only in prose. Without it this corpus would be pure arithmetic and there would be nothing for a model to be measured on at all.

Synthetic Record
----------------------------------------------------------------
This file is invented for a public evaluation kit. It is not a real contract deliverable record.
Every contract number, programme name, CDRL item, data item title, person, office and date in it
is fictional, and it reproduces no DD Form 1423, no Data Item Description, no contract and no
programme's deliverable register. The watch rule reproduced below was invented here and is not an
authority.

Contract
----------------------------------------------------------------
Contract number     : FA8620-24-C-0117
Programme           : SENTINEL RIDGE
Scope               : Airframe structures upgrade
Contractor          : Merrow Aerospace Systems
Administered by     : DCMA Springfield

Deliverable
----------------------------------------------------------------
CDRL item           : A001
Data item title     : Contract Work Breakdown Structure
Recurrence          : One-time
Submission medium   : Government contractor data portal
Register run        : W1 (2026-07-06)

Schedule
----------------------------------------------------------------
Run date            : 2026-07-06
Contract due date   : 2026-10-05
Revised due date    : (none)
Review period       : 30 days

Submission Status
----------------------------------------------------------------
Status              : IN PROGRESS
Last submitted      : (none)
Customer disposition: (none)
Resubmission due    : (none)

Watch Rule
----------------------------------------------------------------
DELIVERABLE WATCH RULE  (illustrative -- reproduces no DFARS clause, no Data Item Description
and no programme CDRL management plan)

The watch runs every Monday at 06:00 local. On each run every deliverable that has not been

The outcomeWhat a good result looks like

A watchlist: for each open deliverable, a band (NONE, WATCH, RAISE, ESCALATE), the date it is being judged against, the days it has spent past its raise gate, and the blocker. A good run raises everything that had to be raised, disturbs nothing that did not, and says which of those answers depended on remembering last week.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• You have a deliverable register and a written lead-time rule, and the notes are written to a house template. — The free rules engine (evals/baseline.py, rules). No provider, no key, no bill.
  It scored 100.00 pct on every field of this corpus. A model cannot beat 100 and costs money to tie it.
• The notes are free prose from many people, and house style changes. — The model arm, with the free floor still computed beside it on every row.
  The free pattern's whole score on the rows that matter is a fact about wording: reword them and it falls from 100.00 pct to 50.00 pct on the band and to 0.00 pct on the blocker.
• You cannot guarantee the watch runs every week. — Monitor the watch itself before you monitor anything with it.
  One skipped Monday put 16 raises a week late, lost 35 days of accrual and took standing-escalation accuracy from 100.00 pct to 63.16 pct — with the reader unchanged.

And where nothing here is good enough:

• You want the raise list to be earlier rather than more accurate. — Neither. Change the CADENCE.
  A 7-day interval against a 5-day raise gate leaves a 2-day band in which a due date arrives with no run ever having seen it inside the gate: 5 of 30 deliverables here are never warned in time on any schedule, by any reader.
• Your register is keyed on something that gets renumbered. — Nothing here yet. Build the identity check first.
  Every carried scalar is keyed on the deliverable id. A re-keyed line arrives with no history and every previous-run rule silently stops applying to it.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-23. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-23 — b003-cdrl-watch-rules. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Clone, run python3 -m tools.build_corpus, and every free arm reproduces from a fixed seed with no key and no network. The recorded results ship in results/, so the kit is browsable offline; python3 -m src.app renders every panel except the model column.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. End to end for ONE deliverable on ONE scheduled run: build the prompt from the selected sections plus the carried state, one completion call, parse the reply. The carried state is computed by pure code before the call and adds no measurable time. Measured over the 120 scored rows of r001-cdrl-watch.

Current processWhat it replaces

The Monday morning pass down a contract deliverable register — thirty open CDRL lines, each with a due date, a submission status and a paragraph of somebody's notes — deciding which ones to put in front of the programme analyst this week and which to leave standing.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• 4 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one deliverables (4 scheduled runs each) is

No split and no chunking. The unit is a DELIVERABLE — one CDRL line read on four consecutive weekly runs, processed strictly in order because run 3's prompt contains state produced by run 2. Each record goes whole into one call, minus the one section no field asks for.

SetupWhat the setup figure measured

There is no index and no retrieval step. One deliverable record goes whole into one call, minus the one section no field asks for.

LicenceLicence

MIT — generated in this repository by tools/build_corpus.py from a fixed seed, so it reproduces byte for byte and belongs to nobody.

Bring your ownBring your own deliverable records

One record is ONE CDRL LINE AS IT STOOD ON ONE SCHEDULED RUN — not one deliverable and not one document. The same line appears again next week as a separate record, and the difference between the two is the whole finding, so a register exported once is a corpus of one run and this kit has nothing to compare it against. Point src/monitor.reading_of at your own register export and rewrite src/register.RULE_TEXT, WATCH_AT_DAYS and RAISE_AT_DAYS to your programme's own lead windows — those three are the first things to change and the shipped values are illustrative. Then run the free rules floor against your own labelled month before you configure a provider at all: on this corpus it does the entire job, and the only measured reason to add a model is that your analysts' prose does not read like a template.

⚠︎ And what stops being true when you do: This kit reads a register and writes a watchlist. It does not read the deliverables themselves, and it must not: the atlas row it was built from marks the tracking metadata as not controlled technical data while noting that the CONTENT of a data item may well be. Keeping the content out of scope is what makes the export safe to send anywhere. If you extend this to read the documents, that boundary is gone and export control becomes your first design constraint, not your last.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 6,588 characters assembled, 4,380 are records — 66% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Both messages, in the order they are sent: the system message, then the user message replayed from the kit's own src/prompt.py::build on CDRL-0018-W3 with the exact carried state the arithmetic produces for that row — the identical code path every arm uses, not a paraphrase. THE SPLIT IS IN CHARACTERS AND THIS LINE SAYS SO: per-part TOKEN counts were not measured, because measuring them means sending nested prefixes of the prompt to a provider and this kit spent no calls on that. The billed input token count in tokens is the run's own average and is not the sum of the parts below.

Full promptThe prompt in full

You run one scheduled pass of a contract deliverable watch. You apply a written watch rule to one deliverable and answer with one JSON object and no other text.

You are running one scheduled pass of a contract deliverable watch. The watch re-reads every open
deliverable on a fixed weekly schedule; you are being shown ONE deliverable as it stands on this
run.

The watch rule, its ladder and the rules that depend on the previous run are reproduced in the
record below. Apply them exactly as written. The rules that depend on earlier runs cannot be
answered from this record: what is known about the previous run is stated under "Carried state" and
is the only history available to you. Do not assume anything about earlier runs beyond it.

Read the Program Notes. They are the only place the record says whether a printed revised due date
is carried by an EXECUTED contract modification or is merely proposed, and that decides which date
this deliverable is judged against.

Answer with a single JSON object and nothing else:

{"raise_band": "NONE|WATCH|RAISE|ESCALATE",
 "governing_clock": "CONTRACT_DUE|REVISED_DUE|RESUBMISSION_DUE|CUSTOMER_REVIEW_SLA|NOT_APPLICABLE",
 "days_at_risk": <whole number>,
 "blocker_cause": "AWAITING_GOVERNMENT_DATA|AWAITING_INTERNAL_APPROVAL|AWAITING_SUBCONTRACTOR_INPUT|IN_CUSTOMER_REVIEW|REWORK_AFTER_REJECTION|RESOURCE_SHORTFALL|NO_BLOCKER",
 "rationale": "one sentence, naming the rule or the gate you applied"}

"raise_band" is the band to REPORT after the rules that depend on the previous run have been
applied, which is not always the band the ladder alone would give.
"days_at_risk" is the running total AFTER this run, carried forward from the previous run's total
as the watch rule describes. It is a whole number of days and it is never negative.

Carried state
----------------------------------------------------------------
The previous run of this watch was on 2026-07-13. This run is 7 days after it. On that run this deliverable was reported RAISE. The dates alone already put it past its raise gate then. The date it was judged against was 2026-07-15. Its submission status was IN PROGRESS. 0 days at risk had been accumulated for it by the end of that run.

Deliverable record, as of this run
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
This file is invented for a public evaluation kit. It is not a real contract deliverable record.
Every contract number, programme name, CDRL item, data item title, person, office and date in it
is fictional, and it reproduces no DD Form 1423, no Data Item Description, no contract and no
programme's deliverable register. The watch rule reproduced below was invented here and is not an
authority.

Contract
----------------------------------------------------------------
Contract number     : W58RGZ-24-C-0088
Programme           : HIGH LANTERN
Scope               : Rotorcraft avionics refresh
Contractor          : Ninebark Aviation
Administered by     : DCMA Fort Barrow

Deliverable
----------------------------------------------------------------
CDRL item           : A003
Data item title     : Software Development Plan
Recurrence          : One-time
Submission medium   : Government contractor data portal
Register run        : W3 (2026-07-20)

Schedule
----------------------------------------------------------------
Run date            : 2026-07-20
Contract due date   : 2026-07-15
Revised due date    : 2026-08-17   (proposed)
Review period       : 30 days

Submission Status
----------------------------------------------------------------
Status              : IN PROGRESS
Last submitted      : (none)
Customer disposition: (none)
Resubmission due    : (none)

Watch Rule
----------------------------------------------------------------
DELIVERABLE WATCH RULE  (illustrative -- reproduces no DFARS clause, no Data Item Description
and no programme CDRL management plan)

The watch runs every Monday at 06:00 local. On each run every deliverable that has not been
accepted is re-read whole, and each one is placed in one band.

Step 1 -- which date governs.
  If the item is ACCEPTED, no date governs.
  If the customer has REJECTED it, the resubmission due date governs.
  If it has been submitted and the customer has it IN REVIEW, the review period governs and the
    date is the submission date plus the review period printed on this record.
  If a revised due date is printed AND a contract modification carrying it has been EXECUTED, the
    revised date governs.
  Otherwise the contract due date governs. A revised date that is only proposed -- a verbal
    agreement, a request in flight, a modification not yet issued -- does NOT govern, and the
    item is judged against the contract date until the modification exists.

Step 2 -- the ladder, against the governing date.
  more than 15 days away            NONE
  15 days away or fewer             WATCH
  5 days away or fewer, or past     RAISE

Step 3 -- what the previous run makes true, applied in this order.
  a. If the previous run raised this item, the ladder no longer says RAISE, and the item has not
     been accepted, report WATCH for one run (RECOVERY_HOLD). An item going quiet is the single
     thing most worth keeping in front of somebody: it went quiet for a reason, and the reason --
     a date that moved, a submission that has not been acknowledged -- is what has to be checked.
  b. Otherwise, if the governing date is not the date the previous run recorded, report at least
     WATCH and record the reason as DATE_MOVED. Somebody moved it and the modification has to
     be seen.
  c. Otherwise, if the ladder says RAISE, the previous run also raised it, and the submission
     status has not changed since that run, report ESCALATE (STANDING_ESCALATION). One raise is a
     reminder; the same raise with nothing having moved is a different message to a different
     person.

Days at risk. The count of days this deliverable has spent past its raise gate, accumulated
across runs. It gains the interval between two runs when the ladder said RAISE on both of them,
holds otherwise, and resets to zero when the item is accepted. It is not on this page: it is
carried from the previous run.

Program Notes
----------------------------------------------------------------
Blocked: the government-furnished interface data we need for section 3 has not been delivered.
Requested from the programme office and still not received. We have asked for relief to
2026-08-17. The request went in on 2026-07-08 and is still with the contracting officer; nothing
has been executed, so the contract date still stands.

Raw responseThe raw response

{"raise_band":"NONE","governing_clock":"CONTRACT_DUE","days_at_risk":0,"blocker_cause":"NO_BLOCKER","rationale":"Contract due date 2026-10-05 is 91 days after the run date, so the ladder gives NONE and no previous-run rules apply on this first watch run."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch a government contract's deliverable before it's late — 120 deliverable records. One model answered, and every answer was then graded Four different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Exact match, per cell, against a key the arithmetic computed. There is no judge model in this kit and that is not a shortcut: three of the four answers are words from a closed list (four bands, five clocks, seven blocker codes) and the fourth is a whole number of days. A judge is for answers whose correctness is a matter of reading; these are matters of equality, and asking a model to grade RAISE == RAISE would add cost, variance and a second thing to be wrong.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-23. The provider that actually ran the calls is kept out of these tables per this estate's naming rule, so nothing here is what was paid. The real spend is recorded in the shared call ledger, not on this page.

Grading unitWhat the grading figure prices

Grading is evals/scoring.py in-process: no key, no model, no network, and it runs over all 120 rows in well under a second. Pricing this ruler in the units of the thing it measures — 'one call per deliverable-run' — would put a dollar figure on something that genuinely costs nothing.

The gradersFour ways to grade

⚑ THIS FLOOR IS THE HEADLINE, NOT THE COMPARATOR. It scores 100.00 pct on the band, 100.00 pct on the clock, 100.00 pct on the day count and 100.00 pct on the blocker over all 120 rows, for $0.00. A model can tie it and cannot beat it. ⚠︎ AND ITS PERFECTION IS PARTLY A FACT ABOUT THE WORDING. The pattern in evals/baseline.py was written in this repository against the sixteen note templates in tools/build_corpus.py, also written in this repository. b004-cdrl-watch-paraphrase-free puts a number on that: the same 24 rows with the prose reworded and not one fact changed, and the pattern falls to 50.00 pct on the band and 0.00 pct on the blocker. Its errors under rewording are all in the SAFE direction — with no marker matched it treats relief as unreal, which over-raises rather than under-raises — but that is a property of how it was written, not a guarantee. ⚠︎ THE PARAPHRASES SHARE THAT AUTHOR TOO. An author writing paraphrases knows exactly which words to avoid, so the probe is an UPPER BOUND on brittleness and not an estimate.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-23. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same one deliverable on one scheduled run, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

⚑ GRADING THIS KIT IS FREE, AND IT IS A MEASURED ZERO RATHER THAN AN UNPRICED ONE. evals/scoring.py is exact comparison in-process: no key, no model, no network. So are all three floors, the missed-run schedule, the stub, the ten-check pre-flight and the free half of the paraphrase probe — every headline figure on this page was produced without calling a provider at all. 332 provider calls stand behind the figures on these pages. Every one of them is behind a figure above: 8 ceiling calibration, 120 scored, 120 the stateless control, 30 the note-blind ablation, 30 the repeat probe (10 rows asked 3 times) and 24 the paid half of the paraphrase probe. NOTHING WAS DISCARDED and no run was re-fired. ⚠︎ The ledger also carries 121 calls this kit was REFUSED earlier the same day, when the shared account hit zero balance mid-run: 402 Payment Required returns no completion and is billed for none, and they appear on the ledger only because src/budget.py records a call before making it — the direction a spend guard should round.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, and specifically provider-side reasoning. The calibration billed 9879 output tokens against 12399 input across 8 calls, and 9040 of the output (91.51 pct) never reached the reply text at all — it was the model working through the ladder, the governing-date decision and the previous-run rules.
• THE NUMBER OF OPEN DELIVERABLES, multiplied by the number of runs. This is the unit that surprises people: the bill is per row PER PERIOD, not per document. A register of 200 open lines watched weekly is 10,400 calls a year, not 200.
• THE CADENCE. Halving the interval to close the blind window doubles the bill exactly. That trade is the design decision this kit exists to make visible, and it is a cost decision before it is an accuracy one.
• INPUT TOKENS, and they are nearly constant. Every record is between 4434 and 4758 bytes by construction and the carried state is six scalars in one paragraph, so input barely moves — 1535 tokens a row. On this kit input is the small half of the bill.

Your volumeWhat it costs at your volume

Linear in rows and linear in runs, with no cliff in between — there is no index to rebuild, no retrieval to widen and no context that grows. Ten times the register is ten times the calls at the same price per call; ten times the frequency is the same. The only non-linearity is the one below.

Model choiceWhy this model

It is the tier the whole series runs on, so the rows compare across use cases. Nothing about this task argues for a bigger one: the free rules engine already scores 100.00 pct, so a more capable model cannot buy accuracy here — only robustness to wording, which is the one thing this kit has not been able to measure on the model side.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

120 call(s) on r001-cdrl-watch, one completion each, one tier.

The codeEvery file, and what it is for

16 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pyCorpus builder and the register export it produces

The population, re-read WHOLE on every scheduled run: 30 open deliverables x 4 weekly runs = 120 records under data/corpus/, 4434 to 4758 bytes each, generated from one seed. Nothing is sliced out and nothing streams — a register is a list, and every open line is on every run. The answer key is the OUTPUT of src/register.replay, never typed, so a hand-written misreading of the rule cannot ride into the score.

# Generate the corpus and the answer key. Deterministic, seeded, no network, no model.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
RULE = "-" * 64
RUN_DATES = [datetime.date(2026, 7, 6), datetime.date(2026, 7, 13),
RUN_LABELS = ["W1", "W2", "W3", "W4"]
CADENCE_DAYS = 7
BANNER = """\

src/register.pyWatch rule and arithmetic — a swap seam

The ladder, the governing-date decision and the three previous-run rules, in pure code. The gold is this module's output and so is every arm's carried state.

You change it to: WATCH_AT_DAYS, RAISE_AT_DAYS, the governing-date order and the three previous-run rules. The shipped numbers are illustrative and reproduce no clause; this is the file a programme office rewrites first.

# The watch rule, and the arithmetic it implies. Pure code, no model.

NONE, WATCH, RAISE, ESCALATE = "NONE", "WATCH", "RAISE", "ESCALATE"
BANDS = (NONE, WATCH, RAISE, ESCALATE)
CONTRACT_DUE = "CONTRACT_DUE"
REVISED_DUE = "REVISED_DUE"
RESUBMISSION_DUE = "RESUBMISSION_DUE"
CUSTOMER_REVIEW_SLA = "CUSTOMER_REVIEW_SLA"
NOT_APPLICABLE = "NOT_APPLICABLE"
CLOCKS = (CONTRACT_DUE, REVISED_DUE, RESUBMISSION_DUE, CUSTOMER_REVIEW_SLA, NOT_APPLICABLE)
AWAITING_GOVERNMENT_DATA = "AWAITING_GOVERNMENT_DATA"

src/cadence.pyCadence — a swap seam

SEAM 3 — the clock. What one run owns that the last did not, what a missed run costs, and the blind window between the raise gate and the run interval. Every figure it produces is free: the cost of a schedule is a property of the schedule.

You change it to: CADENCE_DAYS. It is not a preference: it is set by the shortest gate on the ladder, and blind_window() computes the exposure the current interval leaves. Change one and re-read the other.

# The clock this watch runs on, and what a missed run costs. Pure code, no model.

CADENCE_DAYS = 7
CADENCE_TEXT = "every Monday at 06:00 local, seven days apart"
CADENCE_TRIGGER = ("a scheduler on the machine that holds the register export — cron, a scheduled "
def blind_window(cadence_days=CADENCE_DAYS, raise_at=R.RAISE_AT_DAYS):
def owned_by(run_dates, index, rows_by_unit):
def replay_schedule(run_dates, rows, keep):
def never_warned(run_dates, rows, keep):
def missed_run_cost(run_dates, units, skip):
def next_run(after, cadence_days=CADENCE_DAYS):

src/state.pyCarried state — a swap seam

SEAM 2 — the thing that makes this a monitor. Six scalars per deliverable (band reported, ladder band observed, governing date, submission status, days at risk, run date), written from the arithmetic and never from the model's reply, rendered into English for the prompt. Constant cost on run 400 as on run 4.

You change it to: What is remembered between runs, and how it is put to the model. Six scalars and one English sentence per field. Grow this and the kit stops being cheap on run 400.

# The carried state — the thing that makes this a monitor and not another one-shot classifier.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEFAULT_PATH = os.path.join(HERE, "data", "state.json")
def load(path=DEFAULT_PATH):
def save(store, path=DEFAULT_PATH):
def for_unit(store, unit_id):
def advance(store, unit_id, run_date, row):
def _serialise(st):
def _as_date(v):
def describe(state, run_date=None):

src/segment.pySection splitter

Splits a record into its eight named sections. Pure code.

# Split one deliverable record into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "Contract", "Deliverable", "Schedule", "Submission Status",
RULE = "-" * 64
def split(text):
def render(secs):

src/select.pySection selector — a swap seam

Decides what leaves the machine. Government Points of Contact is mapped by no field and is subtracted unconditionally — not by a fallback that happens never to fire.

You change it to: SECTION_HINTS and NEVER_SENT. A section no field maps to is subtracted unconditionally, so adding a field is also a disclosure decision.

# Pick which sections of a deliverable record are sent. Pure code — the last deterministic step

BANNER = "Synthetic Record"
CONTRACT = "Contract"
DELIVERABLE = "Deliverable"
SCHEDULE = "Schedule"
STATUS = "Submission Status"
RULEBLOCK = "Watch Rule"
NOTES = "Program Notes"
CONTACTS = "Government Points of Contact"
NEVER_SENT = (CONTACTS,)

src/prompt.pyPrompt builder

Three parts: the selected sections, the carried state, the question. The stateless control and the note-blind ablation are this builder with one part replaced or removed, and check_labels asserts they differ nowhere else.

# Assemble the one prompt this kit sends. No template engine, no framework — string concatenation

INSTRUCTION = """\
NO_HISTORY = ("No previous run is available for this deliverable. Judge it on this record alone, "
def build(text, carried, run_date=None, stateless=False, note_blind=False):

src/adapters/__init__.pyModel adapter — a swap seam

SEAM 1 — the model. Raw HTTP for every provider, standard library only. Returns input/output tokens, reasoning tokens and finish_reason beside the text, and separates a transport failure from an HTTP status so a dropped connection is retried rather than recorded as a lost row.

You change it to: PROVIDER, BASE_URL, MODEL in .env. Adding a provider is one function and one entry in PROVIDERS; it must return token counts, because the cost lens prices them.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

evals/baseline.pyFree floors

Three of them: dates only, dates plus a pattern over the prose, and the full rules engine with memory. The third is the real competitor and it wins.

# The free floors. No model, no key, no network — arithmetic and regular expressions.

def blocker_from_structure(row):
def _flat(notes):
def blocker_from_notes(notes, row):
def authority_from_notes(notes):
def resolve(row, use_notes):
def answer(row, run_date, state, use_notes, use_state):
FLOORS = {

evals/scoring.pyScorer

Exact match per cell against the computed gold, split ten ways an average would hide — the four fields, the watchlist, the quiet rows, the relief rows, the standing escalations, the recovery holds and the accruing rows. No judge model.

# Score a run against the computed gold. Pure code, no model, no judge.

FIELDS = ("raise_band", "governing_clock", "days_at_risk", "blocker_cause")
RAISED = ("RAISE", "ESCALATE")
def _pct(n, d):
def _norm(v):
def _int(v):
def score(records, golds):
def _gold_key(field):
def compare(a, b):

evals/check_labels.pyPre-flight

Ten checks that must pass BEFORE a run may spend, including both directions of the never-sent guard, a re-derivation of every gold label, a proof that every reason code is reachable, and a proof that the one off-page fact is off the page.

# Everything that must be true BEFORE a run may spend. Free, no key, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
def check():
def main():

evals/run.pyHarness

30 deliverable chains, four strictly-ordered runs each, 12 concurrent workers. The state advances even for a run whose CALL failed, so one transport error cannot turn into three scored failures.

# Run the watch over the 30 deliverables and write one result file. THIS SPENDS MONEY.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
def load_gold():
GOLDS = load_gold()
def true_row(doc_id):
def stub_complete(cfg, system, user, max_tokens=1024):
def main():

src/monitor.pyRuntime — a swap seam

One deliverable, one scheduled run, one call. Parses every date and status off the record in pure code — the model is never asked to copy a date off a line — and leaves exactly one fact for the reader: whether a printed revised due date is carried by an executed modification. Holds MAX_TOKENS, the published ceiling.

You change it to: reading_of — the regular expressions that lift the dates and statuses off a record. Point the kit at a real register export and this is the function that changes.

# One deliverable, one scheduled run, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = ("You run one scheduled pass of a contract deliverable watch. You apply a written watch "
MAX_TOKENS = 8000
FIELDS = ("raise_band", "governing_clock", "days_at_risk", "blocker_cause")
def documents():
def units():
def load_doc(doc_id):
def _one(text, label, default=""):

evals/paraphrase.pyRewording probe

Re-scores the 24 relief rows with the prose reworded and the facts unchanged, running the free pattern and the model over the identical rows. The only experiment on this kit that separates the two readers at all.

# The paraphrase probe: does the free floor's perfect score survive a rewording?

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
PARA = os.path.join(HERE, "data", "paraphrase.json")
def restated(doc_id, note):
def main():

evals/repeat.pyRepeat probe

Asks the same rows again and reports how many gave a different answer — scored against ITSELF, not against the key. It is what showed this kit's single missed raise to be variance rather than a misreading.

# The repeat probe: ask the same rows again and see whether the answer moves. THIS SPENDS MONEY.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
GOLDS = {r["doc_id"]: r for r in (json.loads(l) for l in open(GOLD, encoding="utf-8") if l.strip())}
def pick(n=10):
def carried_for(doc_id):
def main():

src/app.pyLocal UI

One deliverable, one run, on 127.0.0.1:9006. Shows the carried state verbatim, what this run owns that the last did not, the free floor's answer beside the model's, and which sections left the machine. Renders fully with no key.

# The minimal local UI. Standard library only — python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
PORT = int(os.environ.get("PORT", "9006"))
def _gold():
GOLD_ROWS = _gold()
def _row(doc_id):
def carried_for(doc_id):
class H(BaseHTTPRequestHandler):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 1535 input and 1066 output tokens per one deliverable on one scheduled run, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

One outbound HTTPS call per deliverable per scheduled run, to whichever provider is configured, carrying seven of the record's eight sections. Everything else is local: the register is a directory of text files, the carried state is a JSON file beside them, the answer key and every score are computed in-process. There is no database, no queue, no framework and nothing listening on a socket except the optional local UI, which binds 127.0.0.1.

The key is read from <repo>/.env, then the kit's own .env, then the real environment, and it is never written to a result file, a log line or a screenshot. src/app.py strips the key and the base URL out of any provider error before returning it to the browser. The repository has never held a credential: both .env files are gitignored from the first commit and .env.example carries empty values.

The experimentWhat the record looks like with the withheld section, and what leaves

Every record carries eight sections and seven reach the provider. The local UI prints the list on every row — sent, or WITHHELD — because a page that simply does not mention the contacts cannot be told apart from one that quietly sent them. Both screenshots on this page show that table with the withheld row in it. python3 -m evals.check_labels # free, no key, ten checks including this one. Last run on 2026-08-23, on the committed checkout.

Four gates, all of them code rather than policy. The first three are about one section of one document shape and are testable; the fourth is an absence and is testable by grep. None of them is a model behaviour, which is the point — a guarantee that depends on a model honouring an instruction is not a guarantee.

The resultSeven of eight sections leave the machine. The eighth — named government contacts — is subtracted unconditionally, not by a fallback that happens never to fire.

evals/check_labels.py check 3, run over every record on the committed checkout. It asks two questions of each: does select.sent include anything on NEVER_SENT, and does select.for_field with a field name that maps to nothing include anything on it. Both answer no on all 120.

HonestyWhat this does not prove

• Provider-side retention. What the configured provider keeps, for how long, and whether it trains on it is a property of the account and the contract, not of this code. Nothing here measures it and nothing here can.
• Whether the Program Notes are injectable in practice. The surface is named and no attack was run against it.
• Whether a real register export carries personal data in a section this kit sends. The corpus is generated and its contacts are confined to one section by construction; a real export may put a name anywhere, and SECTION_HINTS would then be a privacy decision as well as a prompt-size one.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

No submission, no relief request, no customer contact — non-configurable. This kit produces a raise band, a governing clock, an accrued day count and a blocker code for a programme analyst to read. It never submits a deliverable, never files an extension request and never sends anything to a government point of contact, and there is no setting that makes it. Separately: the carried state is written by code from the parsed record and the previous state; the model's four answers are scored and are never written back into the history the next run is judged against.

Everywhere and nowhere — it is a property of what is ABSENT. The only writers in the kit are evals/run.py and evals/paraphrase.py (results/*.json), tools/build_corpus.py and tools/paraphrase.py (data/), and src/state.save (data/state.json). src/state.advance() -> src/register.step() is the only thing that touches the carried state, and evals/run.py calls it after every run including one whose call FAILED.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is not a submission tool. It has no portal client, no form, and no upload.
• It is not an approval. A band of ESCALATE is a request for a person's attention, not a decision that anything is contractually late — that is a determination a contracting officer makes.
• It is not a source of truth about dates. Every clock it uses is a field somebody else maintains; the atlas row this kit was built from names that directly, and the kit never invents a date.
• It is not a compliance record. Nothing here is retained, signed or timestamped in a way anybody could rely on later; results/ is evidence about the kit, not about the programme.

WatchedWhat is watched, and why that one

The kit measures 85 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

14 measured by the latest run 71 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

12 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 4 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

ablation · with the model in the path — 2 runs. Columns here are only ever compared with each other.

not a time series No two of these 2 runs measured the same system — they differ on documents, failures, free, max_tokens, rules_stripped, snapshots, thinking — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

triage · with the model in the path — 8 runs. Columns here are only ever compared with each other.

not a time series No two of these 8 runs measured the same system — they differ on accruing_cells, authority_cells, date_moved_cells, documents, max_tokens, provider, recovery_cells, rows_scored, standing_cells, stateless, thinking, times, watchlist_cells, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 2 chips that all say so.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 14 chips that all say so.

DeviationsWhat deviated

0 breaches across 12 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

⚑ THE GUARDRAIL CADENCE AND THE WATCH CADENCE ARE THE SAME CLOCK, and that is unusual enough to say plainly. Every band above is read at the end of each weekly run, because there is a run every week and scoring costs nothing. Three of them are read as a TREND rather than a level, because their level is fine right up until the moment the design has failed: standing_band_pct falling while plain band accuracy holds is a lost state file; input_tokens_total jumping is a record shape or a disclosure change; days_at_risk_when_accruing_pct falling is an interval that has quietly changed length. And one thing is watched that is not a metric at all: WHETHER THE RUN HAPPENED. A missed Monday costs 16 late raises, 35 days of accrual and 36.84 pct of standing-escalation accuracy on this corpus, with the reader unchanged — and a watch that did not run produces no numbers, so every band above is silent about it.

In one lineNo framework, deliberately

No framework and no dependencies at all — argparse, datetime, json, os, random, re, sys, textwrap, urllib, http.client, http.server and concurrent.futures from the standard library. requirements.txt names nothing and says in a comment that the emptiness is load-bearing: a line there means something under src/ or evals/ imports it.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

register export -> segment -> select (one section subtracted) -> prompt (record + carried state + question) -> one completion call -> parse -> score against the computed key; and, in parallel and for nothing, the same record through the free rules engine. The carried state is written by src/register.step from the PARSED record, never from the reply, and hands the next scheduled run six scalars.

The other sideWhat a framework costs you

• No dependency to install, so the fork test is git clone and python3 -m tools.build_corpus.
• No scheduler, so running this weekly is somebody else's cron entry — and the kit measures what it costs when that entry does not fire.
• No eval framework, so the scorer is 200 lines you can read and the graders are four restrictions of one comparison.
• No memory layer, so the state is six scalars a programme office can be shown and can check against its own register.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on b003-cdrl-watch-rules on the free floor, with the carried state and all three rules, 2026-08-23. This kit records telemetry measured per run — 2 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 7 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

5 runs not plotted. b000-cdrl-watch-ladder, b001-cdrl-watch-notes, b002-cdrl-watch-missedrun, b003-cdrl-watch-rules, p002-cdrl-watch-repeat recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the population is re-read whole on each scheduled run, and the prior run's carried state is what makes a reading a CHANGE.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — generated in this repository by tools/build_corpus.py from a fixed seed, so it reproduces byte for byte and belongs to nobody. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe four answered fields, per deliverable-run, exact match against the computed answer key

For each of the 120 deliverable-runs and each of the four answered fields, did the reply equal the computed key? Band, clock and blocker are compared exactly after trimming and upper-casing; the day count is compared as a whole number, and a reply that is not a whole number is a MISS rather than a zero — 0 is a meaningful answer here (it is what a deliverable inside its gate gets) and a parse failure must never be scored as one.

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model. The same function every floor, the stub and the missed-run schedule are scored through.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 120 per field. A row whose reply did not parse counts as a MISS in every field, never as an exclusion.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, computed by src/register.replay over each deliverable's trajectory at generation time. This grader IS the reference, so its own TPR and TNR are not measured — scoring it against itself would be circular.

How tight can the band be? No tuned threshold anywhere on the model path — the reply decides. The ladder's own gates (15 days, 5 days) are declared on every record and are inputs, not tuning. Denominators are printed beside every rate because three of them are small: 24 relief rows, 28 standing escalations, 6 recovery holds.

Cadence: Re-scored on every run of the watch, because scoring is free. In a deployment the key does not exist, so what is watched weekly is the RAISE RATE and the reason-code mix against the previous run: a monitor whose standing-escalation count collapses has usually lost its state file, not improved.

The decisionWhen to reach for it

In one lineMissed raises and false alarms, counted apart

Of the 60 deliverable-runs that had to be raised, how many were reported NONE or WATCH (a MISSED RAISE); and of the 26 quiet rows, how many were disturbed (a FALSE ALARM). Two rates over two different denominators, never averaged.

MethodHow the test was run

evals/scoring.py::score, the same pass as the reference grader.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

missed = raised-in-key but not-raised-in-reply / raised-in-key. false_alarm = quiet-in-key but not-quiet-in-reply / quiet-in-key.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The same computed key. This grader is a re-split of the reference grader's band cells, not a second opinion.

How tight can the band be? No threshold. The partition is on the KEY's band, not on any score.

Cadence: Weekly, with the run. The missed-raise rate is the number a programme office would put on a wall.

The decisionWhen to reach for it

In one lineBand accuracy on the rows a single reading cannot answer

Band accuracy restricted to the 28 deliverable-runs whose correct answer is a standing escalation — the item was already being raised last run, is still past its gate, and its submission status has not moved. Nothing in the record in front of you says any of that.

MethodHow the test was run

evals/scoring.py::score, the same pass.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

hits / 28, over the rows whose key reason is STANDING_ESCALATION.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The same computed key, restricted by the key's own reason code.

How tight can the band be? No threshold.

Cadence: Weekly. This is the row a missed run damages first — the missed-run schedule takes it from 100.00 pct to 63.16 pct while plain band accuracy only falls from 100.00 pct to 92.22 pct.

The decisionWhen to reach for it

In one lineBand accuracy where the record prints a revised due date

Band accuracy restricted to the 24 deliverable-runs whose Schedule block prints a revised due date marked (proposed). Half are carried by an executed contract modification and half are not; the structured page is the same shape either way and only the analyst's prose separates them. This is the only place on this kit where reading earns anything.

MethodHow the test was run

evals/scoring.py::score, the same pass. The paraphrase probe (evals/paraphrase.py) re-scores the identical rows with the prose reworded.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

hits / 24, over the rows where a revised due date is printed.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The same computed key, restricted to rows whose record prints a revised due date.

How tight can the band be? No threshold. The subset is defined by a printed field, not by a score.

Cadence: Weekly, and re-check the WORDING whenever the team that writes the notes changes. A pattern-based reader's accuracy here is a property of house style, not of the task.

The decisionWhen to reach for it