Every store has its own filing deadline, and each franchise agreement counts the clock differently. This app reads each week's filing record, works out if a store is late, and flags stores that keep missing deadlines.
PresenterOpens the private repo. Visible to admins only.
For the royalty compliance teamRestaurants & QSR · Cross-domain
Why it matters
Today's manual process, and the same job with the app
A royalty compliance officer at a multi-store franchise group, reviewing every store's filing status once a week.
✕Today's manual process
1Open the filing log and check each store's own agreement for its deadline shape.
2Work out the deadline manually for that agreement, since each one counts the clock differently.
3Remember past misses from memory to judge if a store is a repeat offender.
4A missed pattern means a habitual late filer keeps going unescalated for weeks.
Every store checked from memory, weekly
✓With the app
1The filing record is read dates, agreement terms and the chase log, all at once.
2The deadline is worked out from that store's own agreement, correctly, every time.
3Repeat offenders are flagged from a running record the app keeps itself, week to week.
4Every store is on one worksheet with its status, its history and what should happen next.
The app checks every store, every week
See it work
One real case, replayed from the app's own recorded run
Store ST-21-P5 filed three days late, and already missed one of its last two reviews.
Catch a franchise's late store filings earlyReference appBuilt to be shaped to your process
5
1This week's record the store, its agreement and the date it filed, read straight off the log.
2What the app found late, and by how many days, worked out from the record.
3Repeat offender, not guessed this store missed one of its last two reviews too.
4What doesn't count no statement was issued here; only the deadline itself starts the clock.
5What happens next a reminder logged, not a bigger step, for a compliance officer to act on.
For engineers
How it is built, and how we measured it
All fourteen steps of the build are written up, from the business case to running it in your own environment.
A small, forkable project that does one job end to end. Run once for real, and every figure on these pages captured from that run.
PresenterOpens the private repo. Visible to admins only.
The business caseThe problem this solves
A franchise group's Royalty & Sales Reporting function has one filing deadline per store per week, and the deadline is not the same shape on every agreement: some tiers count days from the period simply closing, one counts BUSINESS days from HQ's own statement posting, and a handful of legacy agreements count from the store's OWN last filing -- which can be undefined if the store has never filed at all. Past that, whether a chase notice actually went out and whether a store is a repeat offender are facts about what an ops team did, not what a spreadsheet assumes. Today a compliance officer works this out by eye from the raw filing log, one store at a time, on whatever Monday they get to it. A compliance officer opening the store-by-store filing log every Monday, checking each store's own agreement tier for its deadline, and remembering by hand which stores are repeat offenders.
Audience
A franchise ops / royalty compliance team deciding which stores to chase this week and which stores need escalation beyond a reminder. Its own answer is that a written lookup table already does the plain half of this job for free -- which is the point. Every number on these pages came from one real run of this code, not from a vendor page.
The inputThe actual filing-review records
The corpus is 150 filing-review records, 0.69 MB (json 1 · jsonl 1 · txt 150). A FRANCHISOR'S COMPLIANCE LOG NAMES THE STORE'S DISTRICT MANAGER, THEIR PERSONAL MOBILE NUMBER AND EMAIL, and it carries the group's own agreement terms store by store. No franchisor publishes one. The two alternatives were both worse: a scrubbed real log changes what the model is being asked to read, and no corpus at all is a description of a kit. So it is generated and the generator is committed beside it.
⚑ THE SIX AGREEMENT TIERS DISAGREE ON PURPOSE, AND THAT IS THE ONLY DESIGN DECISION IN THE CORPUS THAT MATTERS. FA-1: 5 calendar days from the PERIOD CLOSING. FA-2: 3 calendar days from HQ ISSUING THE STATEMENT. FA-3: 10 calendar days from the PERIOD CLOSING. FA-4: 2 BUSINESS days from HQ ISSUING THE STATEMENT. FA-5: 7 calendar days from the STORE'S OWN PRIOR FILING -- the one anchor HQ does not control. FA-6: nothing on file. "Five days to file" sounds like one rule; five days from WHAT decides whether a store is on time or three days overdue.
⚠︎ THE WINDOW LENGTHS, THE ANCHORS, THE CHRONIC THRESHOLD AND THE ESCALATION LADDER ARE INVENTED AND REPRODUCE NO FRANCHISOR'S REPORTING POLICY. They were written to make the arithmetic checkable and they name nobody. The claim that agreements differ in this way is a claim about the shape of a multi-generation franchise portfolio, not a quotation of anybody's terms.
The corpus
The 150 filing-review recordsgenerated from a fixed seed, so no real record, person or institution appears in it.
Where each came fromwritten for this kit rather than collected — the corpus is generated in the kit's own repository, so there is no third-party data in it.
Swap this folder for your own material and the kit is pointed at your filing-review records. That is the whole change — there is no database to migrate.
One filing-review record, as the model receives itST-01-P1.txt · 1 of 150
Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated filing-compliance review record for
an AI use-case kit. It names no franchise brand, reproduces no franchisor's reporting
policy and describes no real store, district manager or franchisee.
Store
----------------------------------------------------------------
Store code : ST-01 (Northgate)
Agreement tier : FA-1
Franchise region : Northgate
Reporting Facts
----------------------------------------------------------------
Report type : Weekly Sales & Royalty Report
Period end date : 2026-01-11
Statement issued : -- not yet issued as of this review
Prior period filing date : n/a -- this tier's window does not anchor on it
Submitted : 2026-01-12
Agreement Terms
----------------------------------------------------------------
Agreement on file for FA-1 (Tier 1 Agreement):
"The Weekly Sales & Royalty Report is due within 5 calendar days of the reporting period closing."
Rule F-1 Each agreement tier's filing deadline is READ FROM THE TERMS on this record: how long it
is, what unit it is counted in, and which ANCHOR EVENT it runs from. None of the three may
be assumed and none of them is the same on every tier.
Rule F-2 The deadline is the anchor date plus the window, in the unit the terms state. Where the
terms state BUSINESS days, Saturdays and Sundays are not counted, and counting starts on
the day AFTER the anchor date.
Rule F-3 A period is ON_TIME if the report was submitted at or before the deadline; LATE if
Abridged — the file continues.
The outcomeWhat a good result looks like
Every store's current period is on the worksheet with its filing status, how many days late (if any), whether a chase notice was actually logged, whether the store is currently a chronic offender under its own rolling record, and what escalation -- if any -- followed.
And when it cannot
Two directions, and they cost different things. A MISSED chronic flag leaves a repeat offender un-escalated. A FALSE chronic flag spends a compliance officer's attention on a store that did not need it. The scored run made neither (0 missed, 0 false, over the 32 chronic cells) -- its one failure was a dropped reading at the token ceiling, recorded as a failure rather than smoothed over. The stateless control made 32 missed chronic flags out of 32, which is what happens when the mechanism behind the whole second eval question is switched off.
Where it fitsWhat did work
Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.
Your agreement tiers' terms are stable and you have read all of them — the b002 table-mem floor in this repo -- under 200 lines of Python, no key, no bill It scores a cleaner result than the model on every band of this corpus, at $0.00. This kit's own measurement says so and the page leads with it.
You want the chronic-offender list and nothing else about the deadline — the carried-state mechanism itself (src/state.py + a rolling count), whether or not a model reads it The memory ablation is the whole finding: pulling the carried state out drops chronic recall from 100.00% to 0.00% while every other field barely moves. Whatever reads the prompt, it MUST see the rolling history or this call cannot be made at all.
You onboard agreement tiers faster than anyone re-reads their terms, or you run a legacy prior-filing tier — a reader that takes the terms off the record every time, the way this kit's model does, rather than a maintained table This kit did not run a terms-drift experiment (unlike its closest sibling, reimburse-chase, which measured the written table falling to 0.00% the moment one platform's window moved). The risk is real and unmeasured here -- see Eval.could_not_verify.
You need the notice actually sent or the store actually escalated, not listed — a person acting on the worksheet, or a franchise-management system integration this kit is not Nothing in this repository sends a notice, opens a cure-notice workflow or edits an agreement. LATE and CHRONIC are rows on a worksheet.
At a glanceHow the whole thing runs
99%status accuracy pct
5,760 msp50, end to end
$3.80per 1,000 filing-review records · Google Gemini 3 Flash
Run once, for real, on 2026-08-24. Every figure on these pages was captured from that run — nothing is written from intent.
14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.
Catch a franchise's late store filings early14 steps · 4 questions · run once, for real · 2026-08-24
Every tile is a linkall 14 steps · no step without a page
Should you use this?What you bring, where it stops, and when not to use it
Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.
What do I have to bring?
Point tools/build_corpus.py at your own filing log, or replace data/corpus/*.txt and data/gold.jsonl directly. The measured figures on this page do not travel with your corpus, and several of them are corpus properties rather than model properties.Corpus lens →
When is this the wrong choice?
Avoid: AVOID paying a model per store per week to do date arithmetic you can write down. That is the most expensive way to get an answer a dict already has. That is the case against the best-fitting scenario (“Your agreement tiers' terms are stable and you have read all of them”). 4 scenarios scored in all, each with its own.Eval lens →
Where does it stop working?
The corpus is generated, so every field the rule needs is a labelled line in a fixed seven-section record. A real compliance export is whatever shape the franchise-management system produces, and mapping that onto these seven sections is work this kit does not do and does not measure. 7 recorded failure modes, each from a run rather than a guess.Corpus lens →
What was never verified?
WHETHER THIS RESULT IS STABLE ACROSS REPEATS. One run per arm. 9 items this kit says it could not check.Eval lens →
Can I run this on a model I control?
Yes — any OpenAI-compatible endpoint, including one on your own hardware. The shipped adapter takes its host from BASE_URL and its model from MODEL, so nothing in src/ changes. The published figures come from 1 model on the fast tier, with the carried state, one provider, one key. Prompt lens →
And if it fits — what do I stand up?
4 artifacts with a stated home and a stated egress, and 5 decisions each with what you provision past its ceiling — plus what was not measured. That is the next page, not this one.step 14 — Run it in your environment →
Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).
Last verified 2026-08-24 — r001-filing-chase. Every figure on these pages was captured from that run.
Run itHow this reaches your data
Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.
Run this on your own data
The pipeline, its eval harness and the runs behind every numberdeployed inside your environment, on your own model endpoints, against your own documents.
The corpus above is the shape, not the limitit is a folder swap, and there is no database to migrate.
Checked before this shipped — A fresh clone with NO key configured reproduces: the 150 records and the answer key (regenerated from the seed in under a second), all 15 pre-flight assertions, all three free-floor arms scored end to end, the cadence chronic-delay report, and the UI at 127.0.0.1:8206 with the recorded r001 answers replayable off the committed result file. What it cannot reproduce without a key is the two paid arms (r001, s001) -- and their result files are committed, so the numbers are checkable without paying.
PresenterOpens the private repo. Visible to admins only.
Step 01 of 14Business case
Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.
In brief
99.33%rows answered
5,760 msp50, end to end
17,841 msp95
1 minclone to first result
What the clock covers. END-TO-END per reading: one HTTP request carrying the assembled prompt, and the reply parsed to a status, a days-late figure, a notice_sent call, a chronic call and an escalation code. THE TAIL IS THE STORY: p95 (17.8s) is 3.1 times p50 (5.8s). Reply length is driven by how much the reading has to reconcile -- a chronic determination that has to weigh a rolling miss count against an injection-shaped ops note takes longer than an ON_TIME period with nothing to reconcile.
Current processWhat it replaces
A compliance officer opening the store-by-store filing log every Monday, checking each store's own agreement tier for its deadline, and remembering by hand which stores are repeat offenders.
Where it is not good enough
⚑ THE FREE FLOOR HAS A CLEANER RECORD THAN THE MODEL ON THIS SHIPPED CORPUS, AND THE PAGE LEADS WITH THAT. table-mem -- the compliance team's own written table of each tier's window, unit and anchor, plus business-day arithmetic and the same carried history -- scored 100.00% on all five fields over all 150 readings, for $0.00. The model scored 99.33% on four of the five fields and 98.00% on days-late, over 149 of 150: one reply (ST-07-P2) was cut off at the 8,000-token ceiling before it finished, entirely because provider-side reasoning ate the budget (91.1% of this run's output tokens were reasoning nobody asked for). A restaurant franchise reading this page should build the table, not buy the model, for the plain lookup half of this task.
⚠︎ THE TIE IS NOT A FAIR FIGHT IN THE FLOOR'S FAVOUR. evals/baseline.py's table and the answer key are two expressions of ONE rule, so the floor cannot misread it. The model has only the prose in the Agreement Terms block, and it is also the only arm asked to weigh an injection-shaped ops note ('go easy on the chase this period') against the rule -- which it did correctly on every example checked, but that robustness is not what the accuracy table above measures.
⚑ WHERE MEMORY IS THE WHOLE ANSWER. Remove the carried state (s001) and days-late, notice_sent, status and escalation barely move -- but chronic collapses to 78.67% recall, missing ALL 32 chronic-offender cells outright, because a rolling miss count needs a memory to roll. This is the one place the exercise of reading a prompt correctly, rather than a lookup table, is unambiguously required.
⚠︎ AND THE TWO REMAINING DAYS-LATE ERRORS ARE BOTH REAL REASONING SLIPS, NOT NOISE. On ST-17-P6 the model correctly named the deadline and the actual filing date in its own rationale, then computed days_late from the REVIEW date instead of the FILING date (10 instead of 6). On ST-12-P2 the reported number (2) does not even match the two dates the model's own rationale states (which imply 1). Both are quoted in Eval.taxonomy.
⚠︎ WHAT A PERFECT FLOOR SCORE DOES NOT MEAN. Every field the rule needs is a labelled line in a fixed seven-section record, and the notice/escalation facts are rendered as a fixed-format system-log line a regex reads as reliably as a model does. A real compliance export is rarely this tidy -- see Data.breaks_on.
One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.
→Sources
txt150jsonl1json1
25 franchise stores, six agreement tiers, each re-read on 6 weekly reviews — 150 readings
written tier table + carried history: 100.00% on all five fields, $0.00
flat 3-day rule alone (no table, no memory): 50.00% status, 49.33% chronic
Recorded failureone legacy tier anchors a store's own deadline on ITS OWN prior filing — a store that never establishes a filing history reads CONTEXT_INCOMPLETE on every period, forever, which no table can fix
missed and false chronic flags reported apart, never averaged
Recorded failure1 dropped reading of 150 (token ceiling) — and table-mem makes 0, on the identical 150 x 5 cells; the two arms separate on chronic recall alone once memory is removed, 100.00% against 0.00%
It produces a worksheet row for a compliance officer to action — a filing status, how many days late, whether a chase notice actually went out, whether the store is a chronic offender under its own rolling record, and what escalation followed — and it never sends a notice, escalates a store or edits a franchise agreement: evals/check_labels.py greps every .py and .js file in the kit for such a code path and passes at zero. ⚑ TWO STATIONS MAKE THIS A MONITOR AND NEITHER IS THE MODEL. The carried state is two fields written by src/filing.step from the store's own recent statuses and never from a reply — pulled out, chronic recall does not merely fall, it goes to zero, because reaching the chronic threshold always needs at least one miss carried in from an earlier period. The clock is the second: one weekly review owns every store's current period, and a skipped Monday is not reconstructed by arithmetic, only priced by it, for $0.00 (src/cadence.py).
⚠︎ THE FLOOR STATION CARRIES THE FLOOR'S OWN NUMBER, NOT THE MODEL'S, because on this corpus that number BEATS the model's: table-mem — the compliance team's written tier table plus the same carried history — scores 100.00% on all five fields where the model's one dropped reading holds it to 99.33%/98.00%. A franchise operator reading this page should build the table.
⚠︎ AND THE SHARPEST STRUCTURAL FINDING NEVER GOES THROUGH A MODEL AT ALL: one legacy agreement tier anchors a store's deadline on the STORE'S OWN prior filing date rather than on anything HQ controls, and one store on that tier never establishes a filing history at all — every one of its six periods reads CONTEXT_INCOMPLETE, not because its agreement was hidden but because the anchor event the agreement names never occurs. A period-end or statement anchor cannot fail this way, because both are facts HQ always knows.
⚠︎ WHAT A CLEAN FLOOR SCORE ON A GENERATED CORPUS DOES NOT MEAN: every tier's terms here are one labelled sentence a generator wrote, and notice_sent/escalation are both parsed off a fixed-format system-log line every arm reads alike. A real compliance export is messier on both counts, and Data.breaks_on says so rather than hiding it.
⚠︎ NO FRANCHISE BRAND, DISTRICT MANAGER OR STORE IS REAL: the corpus is self-authored synthetic (tools/build_corpus.py, fixed seed), and Store Manager Contact — the DM's name, personal mobile and email — is withheld from every prompt by src/select.py; nothing about a person reaches either figure.
The swap seams
Seam
File
What changes
the model
src/adapters/__init__.py
PROVIDER, BASE_URL, API_KEY and MODEL in .env. Adding a provider is one function and one entry in PROVIDERS.
the carried state
src/state.py
what fields are carried, and the English sentence they render into. src/filing.step is the only writer.
the cadence
src/cadence.py
CADENCE_DAYS and the review lag, and every derived figure moves with it -- the chronic-delay report recomputes rather than restates.
the tier table
evals/baseline.py
TIER_TABLE -- the free floor's written copy of each tier's window, unit and anchor. The MODEL has no such table: it reads the terms off the record.
the sections that are sent
src/select.py
SECTION_HINTS and NEVER_SENT
the corpus
tools/build_corpus.py
the seed, the tier table, the store reliability profiles and the review horizon
Components
Component
File
Role
the corpus generator
tools/build_corpus.py
Generates 25 stores x 6 weekly periods = 150 records from a fixed seed (SEED = 20260824). Each store's six periods are the first six weekly reviews the watch could have seen it, so different stores are read on the same six calendar weeks. A quarter of the stores are drawn from an unreliable reliability profile so the chronic-offender list is a real list. The gold labels are src/filing.step's output over the store model, never typed.
the filing arithmetic
src/filing.py
The rule as pure code: the anchor event this tier's window runs from, the deadline (calendar or business days), the days late, the rolling chronic count, and the precedence between the five statuses. No model, no judgement. ⚠︎ The eight rules, the window lengths, the anchors and the chronic threshold are INVENTED and the kit ships that as a stated limitation -- see Data.why_this_corpus.
the carried state
src/state.py
SEAM 2 -- the thing that makes this a monitor. Two fields per store (the rolling window of recent misses, and the status last reported), written from the arithmetic and never from the model's reply, and rendered into one English sentence for the prompt. It stays small on purpose: a rolling window of at most two prior periods costs the same on review 30 as on review 2.
the clock
src/cadence.py
SEAM 3 -- what a skipped weekly review costs a chronic-offender flag, and it is the only part of this kit that measures something no reading can. Replays the chronic sequence across the six weekly reviews and reports, per week, how many stores newly crossed the threshold and whether a skipped review would still catch them a week late or miss them entirely inside the horizon. Pure date/status arithmetic: no page, no model, no network, $0.00.
the section splitter
src/segment.py
Splits a record into its seven named sections. Pure code; evals/check_labels.py asserts all seven are present in all 150 documents before a run may spend.
the section selector
src/select.py
Decides which sections reach the model. Store Manager Contact -- the district manager's name, personal mobile and email -- is mapped by no field and is subtracted unconditionally by _fallback(), so the one section whose subject is a PERSON never leaves the machine even when a source-system schema change makes every hint match nothing.
the prompt
src/prompt.py
Three parts: the fixed instruction, the carried-state sentence, and the selected sections in document order. The stateless build replaces the sentence with a line saying no history is available -- byte-identical everywhere else, which is what makes the control a control, and evals/check_labels.py asserts exactly that.
the model adapter
src/adapters/__init__.py
SEAM 1 -- the model. Raw HTTP for every provider, standard library only. Returns input/output tokens, reasoning tokens and finish_reason beside the text, and separates a transport failure from an HTTP status so a dropped connection is retried rather than recorded as a lost reading.
the review runtime
src/review.py
One store, one weekly review, one call. Parses the dates off the record with a regex and deliberately does NOT parse the window -- the length, the unit and the anchor are the thing under test. Holds MAX_TOKENS = 8000, borrowed from a sibling monitor kit's own calibration rather than measured here -- see Cost.cost_cliffs, where that borrowing is shown to have cost one dropped reading.
the three free floors
evals/baseline.py
flat3-nomem (one 3-calendar-day window from the period closing for every tier, no memory), flat3-mem (the same wrong rule plus the carried state -- the gap is memory alone), and table-mem (the compliance team's own written tier table plus memory plus an abstention). table-mem has a CLEANER record than the model on the shipped corpus, at $0.00, and that fact is published rather than buried.
the scorer
evals/scoring.py
Exact match per cell against the answer key, in code, with missed and false chronic flags counted apart and never averaged, and days-late accuracy broken out PER AGREEMENT TIER -- because an aggregate hides whether the reader got the ANCHOR right on the tiers that do not anchor on the period simply closing.
the pre-flight
evals/check_labels.py
Fifteen assertions that must hold before a run may spend, each one a claim this page makes. Includes both directions of the privacy guard, the one-variable prompt control, the chronic rolling-window replay from the shipped status sequence, and the assertion that the free table's tier terms agree with the shipped corpus everywhere.
the local UI
src/app.py
One store, one weekly review, its carried state and its verdict, on 127.0.0.1:8206. Renders with no key. It shows the carried sentence verbatim, the free floor's answer beside the model's, a replay of what run r001 actually answered, and the cadence panel -- which needs no key and no model at all.
Where it breaks at scale
⚑ THE CADENCE IS THE SCALING VARIABLE AND IT IS NOT THE CORPUS SIZE. One run of this watch is one call per OPEN STORE-PERIOD in the current filing cycle, and the watch wakes every week. A franchise group with 250 stores pays 250 calls a week, roughly 0.95 USD on the shared projection card, about 3.80 USD a month -- and that is for a task the free table does identically for nothing. There is no batching, no cache and no early exit, because every store is re-read whole on every review and that is what makes a reading a change.
⚠︎ THE LEGACY PRIOR-FILING ANCHOR IS A SECOND, SHARPER SCALING RISK, AND IT IS STRUCTURAL RATHER THAN A VOLUME PROBLEM. A tier whose deadline anchors on the store's OWN last filing can strand a struggling store in CONTEXT_INCOMPLETE indefinitely -- the corpus's own worst-case store never establishes a history and reads CONTEXT_INCOMPLETE on all six of its periods. At scale this means the stores that most need chasing are exactly the ones this tier's arithmetic goes silent on, and nothing in this kit surfaces that as a distinct alert.
⚠︎ AND, AS WITH ANY MONITOR CARRYING STATE, THE STATE FILE IS THE FIRST THING THAT BREAKS UNDER TWO WRITERS. A deployment (not this eval harness, which recomputes state from the answer key) would replace data/state.json atomically -- correct for one writer and not a concurrency model. Two watches over one filing log is two writers, and nothing here has tested it.
PresenterOpens the private repo. Visible to admins only.
Step 03 of 14App UI
Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.
SuccessesWhen it works
ST-21-P5 loaded: a legacy FA-5 store whose window anchors on its OWN prior filing date, with the carried state already showing one recent miss and the Chase & Escalation Log carrying an instruction-shaped ops note ('go easy on the chase this period') before any button is pressed.successOpen full size →The cadence panel, pure arithmetic over all 150 documents' chronic history: 11 stores newly crossed the chronic threshold across six weekly reviews, concentrated in week 3, and 2 of those events would be missed entirely -- not merely delayed -- had that week's review been skipped.successOpen full size →With no API_KEY configured, the read button says so plainly rather than erroring, and the record, the carried state, the free floor and the cadence panel all still render from local computation.successOpen full size →The scored run's real recorded answer for ST-21-P5, replayed for free off results/eval-r001-filing-chase.json: LATE, 3 days late, notice sent, CHRONIC, escalation REMINDER -- agreeing with the free floor on every field, and correctly ignoring the ops note's request not to escalate.successOpen full size →
LimitsWhen it does not
A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.
ST-07-P2, replayed the same way: the page states plainly that run r001-filing-chase recorded no answer for this document, because its reply was cut off at the 8,000-token ceiling before it parsed. The free floor column still computes (LATE, 5 days late, notice sent, not chronic, no escalation) because it never runs out of anything.failureOpen full size →
PresenterOpens the private repo. Visible to admins only.
Step 04 of 14Corpus
Written for: "can I point this at mine?" · Measured off the corpus the run actually used.
In brief
150filing-review records
0.69 MiBjson 1 · jsonl 1 · txt 150
25stores (6 weekly periods each) · p50 6 chars
$0.00setup · 0.0s
How it is cutWhat one stores (6 weekly periods each) is
No split, and no chunking. The unit is a STORE -- six consecutive weekly periods processed strictly in order, because a later period's chronic call depends on a rolling miss count produced by the ones before it. Each record goes whole into one call, minus the one section no field maps to.
SetupWhat the setup figure measured
There is no index and no retrieval step -- the filing log is re-read whole on each weekly review. tools/build_corpus.py writes 150 records, the answer key, the store model and the corpus statistics from a fixed seed in under a second, on a laptop, with no network.
LicenceLicence
MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved; every store, tier, filing date, chase-log entry and district manager is invented by the committed generator.
Bring your ownBring your own filing-review records
Point tools/build_corpus.py at your own filing log, or replace data/corpus/*.txt and data/gold.jsonl directly. Keep the seven section headings in src/segment.py::SECTIONS -- the parser asserts all seven in every record before a run may spend -- and put YOUR agreement tiers' real windows, units and anchors in the Agreement Terms block. Then change evals/baseline.py::TIER_TABLE to match, and re-run evals/check_labels.py: it checks the free floor's table against the shipped terms, which is the failure that would otherwise make your floor look worse than it is.
⚠︎ And what stops being true when you do: The measured figures on this page do not travel with your corpus, and several of them are corpus properties rather than model properties. The cadence figures are per a WEEKLY watch over windows of 2 to 10 days; halve the windows or slow the watch and the losses move, which is why src/cadence.py computes them instead of stating them. And the near-perfect tie with the free table is a property of a corpus whose tier terms match the table exactly and whose notice/escalation facts are machine-readable -- point this at a real, messier compliance log and both of those advantages the floor enjoys here may not hold.
What breaks it
The corpus is generated, so every field the rule needs is a labelled line in a fixed seven-section record. A real compliance export is whatever shape the franchise-management system produces, and mapping that onto these seven sections is work this kit does not do and does not measure.
⚠︎ A PERFECT SCORE ON A GENERATED CORPUS IS A STATEMENT ABOUT THE GENERATOR. The strongest free floor scores 100.00% on all five fields precisely because the tier terms it has hardcoded and the terms the corpus prints agree everywhere -- see Eval.could_not_verify for the drift experiment this kit did NOT run, unlike its closest sibling (reimburse-chase), which did.
⚠︎ NOTICE_SENT AND ESCALATION ARE BOTH PARSED OFF A FIXED-FORMAT SYSTEM-LOG LINE (NOTICE_SENT=YES|NO, ESCALATION=<code>), which is why even the weakest free floor scores 100.00% on both. A real compliance log is rarely this tidy -- a chase notice's record of having been sent is often buried in an email thread or a CRM note, not a machine-parseable line, and nothing here has attempted that harder version.
⚠︎ THE ANCHOR IS STATED IN ENGLISH, ONCE, IN A SENTENCE THE GENERATOR WROTE. A real franchise agreement states its reporting deadline in a paragraph among many, inside a PDF rider. Reading the anchor out of THAT is the hard version of this problem and nothing here has attempted it.
⚠︎ THE CATASTROPHIC PRIOR-FILING CASCADE IS MEASURED ON EXACTLY ONE STORE. Only one of the four FA-5 stores in this corpus draws the worst-case reliability profile and never establishes a filing history; the other three recover within a period or two. Whether the cascade is common or rare on a real legacy-tier portfolio is not measured here.
⚠︎ THE CHRONIC THRESHOLD AND WINDOW (2 misses in the last 3 periods) ARE A SINGLE FIXED VALUE. Nothing here measures how the scores move at a stricter or looser threshold, and a real compliance policy's threshold is unlikely to match this one exactly.
⚠︎ THE CHASE & ESCALATION LOG'S FREE-FORM NOTE IS FOUR INNOCUOUS SENTENCES, ONE MILD INSTRUCTION AND ONE DISTRACTOR. Prose a district manager actually types is longer and more varied; the injection surface is real and is sent deliberately, but the sample of it is small.
PresenterOpens the private repo. Visible to admins only.
Step 05 of 14Prompt
Written for: engineer · Logged from the run, verbatim.
In brief
The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.
AssemblyHow the prompt is assembled
Part
Characters
Tokens
system
217
not measured
instruction
3,344
not measured
carried state
135
not measured
Synthetic Record
322
not measured
Store
203
not measured
Reporting Facts
349
not measured
Agreement Terms
3,070
not measured
Filing Position
285
not measured
Chase & Escalation Log
347
not measured
Total
1,991
This is the cost lesson as arithmetic: of the 8,272 characters assembled, 4,576 are datas — 55% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.
TWO MESSAGES, JOINED IN SEND ORDER. The provider receives a system message and a user message; prompt_verbatim prints both, separated by a blank line, because publishing only the second would be publishing most of a prompt. Replayed from the kit's own src/prompt.build() for ST-21-P5 with the carried state that reading was actually given, rather than logged by the run. What makes the replay checkable is that the section list it produces is the list the run recorded in sections_used -- Synthetic Record, Store, Reporting Facts, Agreement Terms, Filing Position, Chase & Escalation Log -- and that src/prompt.build() is the only thing in the kit that assembles a prompt, so there is no second path a run could have taken. tokens.input is the run's AVERAGE input token count (1991.32, rounded), not logged per reading by this harness -- the same convention its closest sibling (reimburse-chase) uses.
Full promptThe prompt in full
The verbatim prompt, as sent
the prompt, as sent
You are a weekly filing-compliance chase for a franchise group. You apply a written filing rule and one store's own agreement terms to one store at one weekly review. You answer with one JSON object and no other text.
You are the weekly filing-compliance chase for a franchise group's Royalty & Sales Reporting
function. It wakes every week, re-reads every store's current reporting period, and reports each
one. You are reading ONE store at ONE weekly review.
The filing rules and the store's own agreement terms are reproduced in the record below. Apply them
exactly as written. THE FILING DEADLINE IS NOT THE SAME ON TWO AGREEMENT TIERS: its length, the unit
it is counted in and the EVENT IT RUNS FROM are all stated in the Agreement Terms block and none of
them may be assumed from another tier or from a default. You cannot see this store's earlier
reviews; what is known about them is stated under "Carried state" and is the only history available
to you. Do not assume anything about earlier periods beyond it.
How to work it out:
- Read the window length, the unit and the anchor event off the Agreement Terms block.
- Find that anchor's date on this record: the period-end date, the statement-issued date, or the
store's own prior-period filing date, whichever the terms name.
- The deadline is the anchor date plus the window. Where the unit is BUSINESS days, do not count
Saturdays or Sundays, and start counting on the day AFTER the anchor date (Rule F-2).
- A period is ON_TIME if filed at or before the deadline; LATE if filed after it; MISSING if
nothing has been filed and this review's date is past the deadline; NOT_DUE if this review's
date has not yet reached the deadline, or the anchor event itself has not yet occurred
(Rule F-3).
- days_late is whole calendar days from the deadline to the later of the filing date or this
review's date. It is 0 on an ON_TIME period, and null on NOT_DUE or CONTEXT_INCOMPLETE
(Rule F-4).
- notice_sent is YES only when the Chase & Escalation Log records an ACTUAL notice event for THIS
period. A tier's policy on when a notice should go out is not evidence one did (Rule F-5).
- chronic is YES when this store's LATE-or-MISSING periods in its available trailing record --
this period plus what the Carried State says about the periods before it -- reach the threshold
named in Agreement Terms. It is a rolling read of the recent record, not a permanent label
(Rule F-6).
- escalation is whatever the Chase & Escalation Log actually records as this period's escalation
action, or NONE if nothing is logged -- never inferred from chronic status alone (Rule F-7).
- If the agreement terms are not on file, or the anchor timestamp the terms name is missing from
this record, the period is CONTEXT_INCOMPLETE: days_late and chronic null, notice_sent NO,
escalation NONE. Never assume a default window (Rule F-8).
Answer with a single JSON object and nothing else:
{"status": "CONTEXT_INCOMPLETE|NOT_DUE|ON_TIME|LATE|MISSING",
"days_late": <whole calendar days past the deadline, 0 if on time, null if not determinable>,
"notice_sent": "YES|NO",
"chronic": "YES|NO|null",
"escalation": "NONE|REMINDER|FBC_CALL|CURE_NOTICE",
"rationale": "one sentence, naming the anchor you used and the deadline you reached"}
Precedence for "status", applied in this order: CONTEXT_INCOMPLETE beats everything; then NOT_DUE
if the anchor event has not occurred or the deadline has not yet been reached with nothing filed;
then ON_TIME, LATE or MISSING by Rule F-3.
Carried state
----------------------------------------------------------------
Of this store's last 2 resolved period(s) before this one, 1 was/were LATE or MISSING. The previous review reported this store ON_TIME.
Store filing record
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated filing-compliance review record for
an AI use-case kit. It names no franchise brand, reproduces no franchisor's reporting
policy and describes no real store, district manager or franchisee.
Store
----------------------------------------------------------------
Store code : ST-21 (Fernvale)
Agreement tier : FA-5
Franchise region : Fernvale
Reporting Facts
----------------------------------------------------------------
Report type : Weekly Sales & Royalty Report
Period end date : 2026-02-08
Statement issued : -- not yet issued as of this review
Prior period filing date : 2026-02-04
Submitted : 2026-02-14
Agreement Terms
----------------------------------------------------------------
Agreement on file for FA-5 (Tier 5 Agreement (legacy)):
"The Weekly Sales & Royalty Report is due within 7 calendar days of the date THIS STORE filed its immediately preceding period's report."
Rule F-1 Each agreement tier's filing deadline is READ FROM THE TERMS on this record: how long it
is, what unit it is counted in, and which ANCHOR EVENT it runs from. None of the three may
be assumed and none of them is the same on every tier.
Rule F-2 The deadline is the anchor date plus the window, in the unit the terms state. Where the
terms state BUSINESS days, Saturdays and Sundays are not counted, and counting starts on
the day AFTER the anchor date.
Rule F-3 A period is ON_TIME if the report was submitted at or before the deadline; LATE if
submitted after the deadline; MISSING if nothing has been submitted and this review's
date is past the deadline; NOT_DUE if this review's date has not yet reached the deadline
and nothing has been submitted, or the anchor event itself has not yet occurred.
Rule F-4 days_late is the whole number of calendar days from the deadline to the later of the
submission date (if filed) or this review's date (if not filed). It is 0 on an ON_TIME
period. It is null on NOT_DUE and on CONTEXT_INCOMPLETE.
Rule F-5 notice_sent is YES only when the Chase & Escalation Log records an actual notice event
for THIS period. A tier's stated policy on when a notice SHOULD go out is not evidence
that one DID, and nothing may be inferred from lateness alone. It is NO on a period that
is ON_TIME, NOT_DUE or CONTEXT_INCOMPLETE, whatever the log says about other periods.
Rule F-6 chronic is YES when the store's LATE-or-MISSING periods in its available trailing record
-- up to the last 3 periods including this one -- number 2 or more. It is a ROLLING
determination, reassessed fresh every period, not a one-way flag: a store that improves
rolls out of it as older misses age out of the window. It is null on NOT_DUE and on
CONTEXT_INCOMPLETE, because this period's own outcome is not yet known and cannot be
folded into the window either way.
Rule F-7 escalation is the escalation action actually logged for THIS period in the Chase &
Escalation Log -- NONE, REMINDER, FBC_CALL or CURE_NOTICE -- and NONE where nothing is
logged. It is never inferred from chronic status alone: a chronic store with nothing
logged this period is chronic with no escalation, and the record must say so rather than
assume one followed.
Rule F-8 Where the agreement tier's terms are not on file for a store, or the anchor timestamp
that tier's window runs from is missing from the record, the period is
CONTEXT_INCOMPLETE: no window may be assumed, days_late is null, chronic is null,
notice_sent is NO and escalation is NONE.
Filing Position
----------------------------------------------------------------
Review date : 2026-02-20 (reporting period 5 of this store's history)
Watch cadence : every 7 days (Monday 06:00 local)
Previous review : 2026-02-13
Chase & Escalation Log
----------------------------------------------------------------
System notice log for this period:
2026-02-13 NOTICE_SENT=YES tier=1 (auto-email reminder)
ESCALATION=REMINDER -- logged against this store this period.
Ops note: This store's DM asked that we go easy on the chase this period -- new GM still ramping up.
Raw responseThe raw response
The raw response, before any parsing
the response, unparsed
{
"status": "LATE",
"days_late": 3,
"notice_sent": "YES",
"chronic": "YES",
"escalation": "REMINDER",
"rationale": "Using the prior period filing date of 2026-02-04 as the anchor, the 7-calendar-day deadline was 2026-02-11, and the report submitted on 2026-02-14 was after it."
}
PresenterOpens the private repo. Visible to admins only.
Step 06 of 14Evals
Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.
In brief
Catch a franchise's late store filings early — 150 filing-review records. One model answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.
The rulerHow it was graded
Every cell is compared for exact equality against a gold set computed by tools/build_corpus.py from the STORE MODEL -- the anchor dates, the tier's window and the reliability profile -- and its chronic arithmetic re-derived from the shipped status sequence by evals/check_labels.py before any run may spend. No model grades anything and no judge is involved. days_late is compared as a whole number; chronic is compared with null matching only null -- a reply that guesses YES/NO where the key says the period is not yet determinable is a MISS, because reporting a chronic call on a period whose own outcome is not known is the exact failure Rule F-6 exists to prevent.
150filing-review records
150source documents
1model tier
1grading method
MeasurementsWhat was measured
COUNTED149 · 149 / 150status accuracy pct — readings, with the carried stateDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
COUNTED147 · 150 / 150days late accuracy pct — readings, with the carried stateDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
COUNTED103 / 106days late accuracy on determinable pct — readings whose deadline can be computed at all -- the harder scope, and the one the prose leads withDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
COUNTED149 · 150 / 150notice sent accuracy pct — readings, with the carried stateDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
COUNTED149 · 118 / 150chronic accuracy pct — readings, with the carried stateDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
COUNTED149 · 150 / 150escalation accuracy pct — readings, with the carried stateDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
COUNTED0 · 32 / 32missed chronic pct — readings whose correct chronic answer is YES, with the carried stateDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
COUNTED0 / 74false chronic rate pct — readings whose correct chronic answer is NO, with the carried stateDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
COUNTED32 · 0 / 32memory chronic accuracy pct — memory-dependent readings -- chronic cannot be reached from the record aloneDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
COUNTED40 / 40context incomplete recall pct — readings whose tier terms or anchor timestamp are missing -- the guardrail, scoredDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
COUNTED16 / 16missing recall pct — readings whose correct status is MISSINGDecided by == against a fixed gold value. No model graded this and it reproduces to the digit — which makes it checkable, not necessarily right: it is only as good as the gold behind it.
NOT YET KNOWN—A person confirmed the gold is rightThe grader is == and needs no confirming. The gold does: it is machine-derived from the registry's structured modules, and every rate on this page rests on that derivation being faithful to what the prose actually says. Nobody has read a document against its gold record by hand.
Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.
How the method was validated
The METHOD is validated by construction and the construction is asserted: evals/check_labels.py re-derives chronic for all 25 store chains from nothing but the shipped status sequence and requires it to match the committed gold, asserts the corpus carries three different anchor events and two different window units (a days-late figure measured on a corpus where every tier anchors the same way measures nothing), and asserts the free table agrees with the shipped tier terms everywhere. All 15 pass; the run refuses to spend if any does not.
933.16output tokens · the fast tier, with the carried state · 5,760 ms p50
877.28output tokens · the same tier, WITHOUT the carried state (the control) · 5,728 ms p50
Two tiers, one API key
Not two vendors and not a vendor comparison — which is the whole reason the two-model rule could be met without signing up to a second provider. Both tiers send the same retrieved context; the slower one writes about twice as much back, waits 1.0× as long, and lands one row apart on 150. This set does not separate them on quality and no ranking should be read from it — which is not an opinion about the models but something the second run below demonstrated.
Grading costWhat it costs
Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.
Priced at
Per 1M in / out
One filing-review record
1,000 filing-review records
Share that is the prompt
Google Gemini 3 Flash the shared card every kit in this series projects onto, so the rows compare across use cases rather than across price lists
$0.50 / $3.00
$0.003795
$3.80
26%
OpenAI GPT-5.6 Luna the cheapest row on the published set, for the low end of the range
$0.20 / $1.20
$0.001518
$1.52
26%
Anthropic Claude Fable 5 a frontier tier, for the other end of the range
$10.00 / $50.00
$0.066571
$66.57
30%
Same work, 44× the bill
The same filing-review records, the same tokens — only the rate card changed. And across all 3 cards between 26% and 30% of what you pay is the prompt this pipeline sends, not the answer it writes.
THE CADENCE IS THE ONLY KNOB THAT MOVES THE BILL BY A WHOLE MULTIPLE -- but on THIS kit the honest lever is not running the model at all. The free table floor produces a CLEANER record than the model, for $0.00, on the shipped corpus. The question a reader should actually be answering is not 'which model' but 'how confident am I that my agreement tiers' written terms are current' -- and that question is exactly the one this kit did NOT price, unlike its closest sibling (reimburse-chase), which ran a terms-drift arm. See Eval.could_not_verify.
Rates checked 2026-08-18. The provider that actually ran both paid arms is kept out of these tables per this estate's naming rule, so nothing here is what was paid. The real spend for this kit's whole build is on the shared call ledger, not on this page.
Grading unitWhat the grading figure prices
freegrading cost, as measured
The GRADER costs nothing: evals/scoring.py makes no model call, and neither do the three free floors it is compared against, the 15 pre-flight assertions, or the cadence chronic-delay report. The cost of the RUN is a different figure and lives in the Cost lens.
The gradersOne way to grade, and why it is the only one
⚑ THE FREE FLOOR HAS A CLEANER RECORD THAN THE PAID MODEL, ON EVERY BAND. Status 100.00 against 99.33. Days-late 100.00 against 98.00. Notice_sent, chronic and escalation all 100.00 against 99.33. The one cell that separates them is ST-07-P2, cut off at the token ceiling -- the floor never runs out of anything. THE HONEST RECOMMENDATION IS THE TABLE.
The two weaker floors say what each half is worth. flat3-nomem -- one flat 3-calendar-day window from the period closing for every tier, no memory -- scores 50.00% on status and 24.67% on days-late, because it is wrong about the anchor on three of six tiers. Adding ONLY memory (flat3-mem) leaves status and days-late untouched (memory has nothing to do with the window) but takes chronic from 49.33% to 67.33%, because even a WRONG per-period miss signal still carries some real pattern. Adding the per-tier table takes days-late from 24.67% to 100.00% and chronic the rest of the way to 100.00%. So on this task memory buys part of the CHRONIC call and the table buys the WINDOW -- and together they beat the model, for nothing.
the fast tier, with the carried state 99.3% status accuracy · the fast tier, memory removed (THE CONTROL) 99.3% status accuracy · table-mem, the compliance team's own written tier table, FREE 100.0% status accuracy · flat3-mem, one wrong flat window for every tier, plus memory, FREE 50.0% status accuracy · flat3-nomem, the same wrong window, no memory, FREE 50.0% status accuracy · 4 more measured on each run
The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.
Why this set cannot separate them
This labelled set CAN tell the arms apart on chronic, and cannot tell the model from the strongest floor at all. Five arms scored through one module land at 99.33 / 99.33 / 100.00 / 50.00 / 50.00 pct on status (model, stateless control, table floor, flat+memory, flat) and at 99.33 / 78.67 / 100.00 / 67.33 / 49.33 pct on chronic -- and the two orderings are DIFFERENT in the useful way: the control that loses nothing on status loses 20.66 points of chronic, and the flat floors that lose half of status still keep some chronic signal from contaminated memory. ⚠︎ WHAT THIS SET CANNOT SEPARATE is the model from the table floor on four of five fields: the floor scores at or above the model everywhere, so this set has no power to show the model earning anything over a maintained table -- which is exactly the finding, not a limitation of the measurement.
Choosing oneWhen to reach for the model at all
The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.
If your situation is
Use
Because
And avoid
Your agreement tiers' terms are stable and you have read all of them
the b002 table-mem floor in this repo -- under 200 lines of Python, no key, no bill
It scores a cleaner result than the model on every band of this corpus, at $0.00. This kit's own measurement says so and the page leads with it.
AVOID paying a model per store per week to do date arithmetic you can write down. That is the most expensive way to get an answer a dict already has.
You want the chronic-offender list and nothing else about the deadline
the carried-state mechanism itself (src/state.py + a rolling count), whether or not a model reads it
The memory ablation is the whole finding: pulling the carried state out drops chronic recall from 100.00% to 0.00% while every other field barely moves. Whatever reads the prompt, it MUST see the rolling history or this call cannot be made at all.
AVOID assuming a bigger model fixes a stateless design. The failure here is architectural (no history in the prompt), not a capability gap a stronger model would close.
You onboard agreement tiers faster than anyone re-reads their terms, or you run a legacy prior-filing tier
a reader that takes the terms off the record every time, the way this kit's model does, rather than a maintained table
This kit did not run a terms-drift experiment (unlike its closest sibling, reimburse-chase, which measured the written table falling to 0.00% the moment one platform's window moved). The risk is real and unmeasured here -- see Eval.could_not_verify.
AVOID assuming the model catches a terms change you have not put in front of it, or that the free table's clean record here would survive a stale line.
You need the notice actually sent or the store actually escalated, not listed
a person acting on the worksheet, or a franchise-management system integration this kit is not
Nothing in this repository sends a notice, opens a cure-notice workflow or edits an agreement. LATE and CHRONIC are rows on a worksheet.
AVOID reading this kit's output as an action. It is decision support, and the guardrail that keeps it that way is the absence of a code path, not a setting.
This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.
Failure causesHow it fails
Cause
What it means
Rows
A real example
TOKEN_CEILING_CUTOFF
the reply was cut off before it produced parseable JSON
1
ST-07-P2, r001: the reply hit exactly 8,000 of 8,000 output tokens (finish_reason=length) and never closed its JSON object. MAX_TOKENS=8000 was borrowed from a sibling kit's own calibration on a comparable reply shape rather than measured on this corpus --…
DATE_BASIS_CONFUSION_ON_LATE
days_late computed from the review date instead of the actual filing date, on an already-filed LATE period
1
ST-17-P6: the model's own rationale correctly names the deadline (2026-02-17) and the actual filing date (2026-02-23, six days later) -- then reports days_late as 10, which is the review date (2026-02-27) minus the deadline, not the filing date minus the…
ARITHMETIC_INCONSISTENT_WITH_OWN_RATIONALE
the reported day-count does not match the two dates the model's own rationale states
1
ST-12-P2: rationale states a deadline of 2026-01-28 and a submission of 2026-01-29 -- one day apart -- and reports days_late as 2, where its own stated dates imply 1.
CHRONIC_MISS_WITHOUT_MEMORY
the stateless control cannot make the chronic call at all
32
s001, every one of the 32 chronic=YES cells: the record alone never says how many of a store's recent periods were late or missing, so a rolling-window rule reduces to 'assume zero history' and never fires. 0 of 32 caught, 0 false alarms -- not noisy, just…
WRONG_TIER_ANCHOR_ASSUMED
one flat window rule applied to six different anchors
264
b000 flat3-nomem, 264 wrong cells over 150 readings -- 75 on status, 113 on days-late, 76 on chronic (contaminated by the wrong status history). Every FA-3 reading (10-calendar-day window) is judged against an assumed 3-day window and every FA-5 reading…
What we could NOT verify
WHETHER THIS RESULT IS STABLE ACROSS REPEATS. One run per arm. A single run cannot distinguish 'this task is close to deterministic for this model' from 'this run was lucky' (or unlucky, on the one dropped reading), and no repeat probe was fired.
WHETHER ANOTHER MODEL AVOIDS THE TOKEN-CEILING DROP OR THE TWO REASONING SLIPS. One tier was run. No cross-tier comparison exists anywhere on this page.
WHAT A TIER'S TERMS ACTUALLY CHANGING MID-QUARTER WOULD COST. Unlike its closest sibling (reimburse-chase), which ran a 21-call terms-drift experiment and found the free table's window accuracy collapse from 100% to 0% on the drifted platform, this kit did not run the equivalent experiment. The free floor's clean record here is untested against a stale table.
WHETHER RENDERING THE CARRIED STATE AS ENGLISH BEATS RENDERING IT AS JSON. src/state.py argues for English and says in its own docstring that the argument is unmeasured. The experiment costs one more 150-call run and has not been paid for.
WHETHER THE MODEL READS A REAL FRANCHISE AGREEMENT. Every terms block in this corpus is one sentence written by the generator. Extracting a window, a unit and an anchor from an actual agreement rider is the hard version of this problem and nothing here attempts it.
WHETHER THE INSTRUCTION-SHAPED OPS NOTE WOULD MATTER IF IT WERE STRONGER. It is one mild sentence and it moved nothing on the examples checked. A note engineered harder against Rule F-6 was not written and not tried.
WHETHER A REAL COMPLIANCE LOG'S NOTICE/ESCALATION RECORDS WOULD STILL BE THIS EASY TO PARSE. Every notice and escalation fact here is a fixed-format machine-readable line; a real ops team's record of having chased a store is often prose in an email thread, and nothing here has attempted extracting the fact from THAT.
CONCURRENCY. A deployment's data/state.json would be replaced atomically, which is correct for one writer. Two watches over one filing log is two writers and nothing here has tested it.
WHETHER 12 CONCURRENT WORKERS CHANGES ANY ANSWER. Both paid arms ran at 12; nothing was run serially to check that the concurrency is answer-neutral, though the periods within one store are strictly serial by construction.
PresenterOpens the private repo. Visible to admins only.
Step 07 of 14Unit cost
Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.
In brief
Tier
Input tokens
Output tokens
p50 latency
Google Gemini 3 Flash
OpenAI GPT-5.6 Luna
Anthropic Claude Fable 5
the fast tier, with the carried state
1,991.32
933.16
5,760 ms
$0.003795
$0.001518
$0.066571
the same tier, WITHOUT the carried state (the control)
1,965.36
877.28
5,728 ms
$0.003615
$0.001446
$0.063518
The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.
Identical work in both rows — the same reading (one store, at one weekly review), the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.
Answering vs gradingAnswering and grading are two separate bills
300 live calls were attempted for this kit: 150 scored (r001, 149 answered, 1 cut off at the token ceiling) and 150 the stateless control (s001, 150 answered). No calibration run was fired -- MAX_TOKENS was borrowed from a sibling kit's calibration instead, and Cost.cost_cliffs shows what that borrowing cost. The three free floors, all 15 pre-flight assertions and the cadence chronic-delay report cost $0.00 and called nothing.
Cost driversWhat actually moves the bill
OUTPUT TOKENS, AND MOST OF THEM ARE REASONING. 91.1% of this run's output (126,734 of 139,041) was provider-side reasoning left at the default, and output is the larger half of the projected bill on the shared card. Most of what this kit pays for is the model doing date arithmetic that src/filing.py does for nothing.
THE RECORD, WHICH IS FIXED. Every record is within 4% of every other (4,788 to 4,973 bytes) because they are generated to one shape, so input is nearly constant at 1991 tokens a reading. Input is the smaller half of the bill.
THE RULE BLOCK, WHICH IS SENT ON EVERY SINGLE READING. src/filing.RULE_TEXT is 2,791 characters and rides inside the Agreement Terms section of all 150 prompts. It is the largest thing in the prompt that never changes, and caching it is the obvious saving this kit does not implement.
THE CADENCE, WHICH MULTIPLIES EVERYTHING. One reading is one store on one weekly review. A weekly watch over 250 stores is 250 calls a week and roughly 3.80 USD a month on the shared card.
Your volumeWhat it costs at your volume
LINEAR IN STORES x PERIODS, AND THAT IS THE WHOLE WARNING. Ten times the stores is ten times the calls at the same cadence -- there is no batching, no cache and no early exit, because every store is re-read whole on every review. 250 stores on a weekly watch is roughly 3.80 USD a month on the shared card; 2,500 stores is roughly 38 USD a month. ⚠︎ AND THE COMPARISON THAT MATTERS AT 10x IS NOT AGAINST A BIGGER MODEL, IT IS AGAINST ZERO: the free table costs the same at 2,500 stores as at 25, because it is a dict lookup and a date subtraction.
Where pricing changes shape
THE OUTPUT CEILING WAS BORROWED, NOT MEASURED, AND THE BORROWING COST A READING. MAX_TOKENS = 8000 was set by analogy to a sibling monitor kit's own calibration (a comparable reply shape, same model family) rather than calibrated fresh on this corpus. The scored run's largest reply hit the ceiling EXACTLY -- 8,000 of 8,000 output tokens on ST-07-P2 -- and never produced parseable JSON. Unlike a kit with comfortable measured headroom, this cliff is real and it already cost one dropped reading out of 150.
Provider-side reasoning defaults. 91.1% of output was reasoning nobody asked for, so a provider whose default differs reprices this kit without changing anything a reader can see. thinking was never sent and what disabling it does to the answers is not measured.
Your return, with your numbers
Volumeopen store-periods per weekly review -- this run judged 150 (25 stores x 6 weekly periods) per arm, on a one-week watch
What it replacessomebody opening the filing log on whatever Monday they get to it, checking each store's own agreement tier for its deadline, working out which anchor event applies, counting the days (skipping weekends on one tier), reading the chase log for whether a notice actually went out, and remembering by hand which stores are chronic
Time saved per itemnot measured here -- and on this kit the honest ROI question is not model-versus-person but table-versus-person, because the free table scored better than the model
We publish the inputs, not a return: a return depends on your labour cost and your volume. The ROI dashboard takes it from here, and AI Costing explains the math.
Model choiceWhy this model
It is the tier the shared connection was already pointed at, and the kit's whole claim is that swapping it is one line of .env and one more run. No comparison across tiers was attempted. ⚠︎ ON THIS KIT THAT MATTERS LESS THAN USUAL: the free table already has a CLEANER record than this tier on every band, so the interesting unmeasured question is not whether a bigger model does better -- there is no headroom above a floor that already beats it -- but whether a model of ANY size reliably stays under a properly-calibrated token ceiling on this corpus's hardest reasoning chains. That experiment was not run; see Cost.cost_cliffs.
Other modelsOn other models
This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.
296,707input tokens · this run
139,041output tokens
—not priced — no committed card for the provider that ran it
The exact work behind every model figure on these pages: 150 readings, one completion call each, one tier. The 150-call stateless control is priced separately under Cost.cost_of_evaluation_usd.
Model
Provider
One eval pass
This whole run
Per 1,000 queries
Rates as of
gpt-5-6-luna
OpenAI
$0.226
$0.226
$1.52
2026-09-12
gemini-3-flash
Google
$0.565
$0.565
$3.80
2026-09-18
gemini-3-8-flash
Google
$0.744
$0.744
$4.99
2026-09-18
llama-5
Meta
$0.962
$0.962
$6.46
2026-09-18
claude-haiku-4-5
Anthropic
$0.992
$0.992
$6.66
2026-09-12
grok-4-5
xAI
$1.428
$1.428
$9.58
2026-09-18
grok-4-6
xAI
$1.428
$1.428
$9.58
2026-09-18
claude-sonnet-5
Anthropic
$1.984
$1.984
$13.31
2026-09-12
gemini-3-1-pro
Google
$2.262
$2.262
$15.18
2026-09-18
gpt-5-6-terra
OpenAI
$2.262
$2.262
$15.18
2026-09-12
gpt-5-6-sol
OpenAI
$3.968
$3.968
$26.63
2026-09-12
claude-opus-4-8
Anthropic
$4.960
$4.960
$33.29
2026-09-12
claude-opus-5
Anthropic
$4.960
$4.960
$33.29
2026-09-12
claude-fable-5
Anthropic
$9.919
$9.919
$66.57
2026-09-18
claude-fable-5-1
Anthropic
$9.919
$9.919
$66.57
2026-09-18
gpt-6-astra
OpenAI
$9.919
$9.919
$66.57
2026-09-17
Read this against the numbers above
NOT MEASURED. Not one of these rows was run. They are this run's measured token counts multiplied by a published rate, as of each row's own rates_as_of date.
⚑ 91.1 PCT OF THIS RUN'S OUTPUT TOKENS WERE PROVIDER-SIDE REASONING (126,734 of 139,041), left at the provider's default, so every row below prices a reasoning-on workload.
⚑ EVERY ROW PRICES ONE READING, AND A DEPLOYMENT DOES NOT BUY ONE READING. This watch wakes every week and re-reads every open store-period each time, so the bill is rows x reviews. Multiply any row below by your own store count and by 4 for a month.
⚑ AND THE ONLY ROW A READER SHOULD ACT ON IS THE ONE THAT IS NOT IN THIS TABLE: $0.00, for the free tier table that had a CLEANER record than every row below on the shipped corpus. This table prices the model against other models. This kit's own measurement prices it against nothing at all, and nothing at all wins.
PresenterOpens the private repo. Visible to admins only.
The codeEvery file, and what it is for
13 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.
Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.
The files on this page are not in the public GitHub catalog.
tools/build_corpus.pythe corpus generator — a swap seam
Generates 25 stores x 6 weekly periods = 150 records from a fixed seed (SEED = 20260824). Each store's six periods are the first six weekly reviews the watch could have seen it, so different stores are read on the same six calendar weeks. A quarter of the stores are drawn from an unreliable reliability profile so the chronic-offender list is a real list. The gold labels are src/filing.step's output over the store model, never typed.
You change it to: the seed, the tier table, the store reliability profiles and the review horizon
tools/build_corpus.py
# Generate the corpus and the answer key. Deterministic: one seed, one corpus, forever.
HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
STORES_JSON = os.path.join(HERE, "data", "stores.json")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
SEED = 20260824
STORE_COUNT = 25
PERIODS = 6
RULE = "-" * 64
src/filing.pythe filing arithmetic
The rule as pure code: the anchor event this tier's window runs from, the deadline (calendar or business days), the days late, the rolling chronic count, and the precedence between the five statuses. No model, no judgement. ⚠︎ The eight rules, the window lengths, the anchors and the chronic threshold are INVENTED and the kit ships that as a stated limitation -- see Data.why_this_corpus.
src/filing.py
# The filing-compliance rule as arithmetic. Pure code, standard-library dates, no model.
CONTEXT_INCOMPLETE = "CONTEXT_INCOMPLETE"
NOT_DUE = "NOT_DUE"
ON_TIME = "ON_TIME"
LATE = "LATE"
MISSING = "MISSING"
STATUSES = (CONTEXT_INCOMPLETE, NOT_DUE, ON_TIME, LATE, MISSING)
YES = "YES"
NO = "NO"
NOTICE_CALLS = (YES, NO)
src/state.pythe carried state — a swap seam
SEAM 2 -- the thing that makes this a monitor. Two fields per store (the rolling window of recent misses, and the status last reported), written from the arithmetic and never from the model's reply, and rendered into one English sentence for the prompt. It stays small on purpose: a rolling window of at most two prior periods costs the same on review 30 as on review 2.
You change it to: what fields are carried, and the English sentence they render into. src/filing.step is the only writer.
src/state.py
# The carried state -- the thing that makes this a monitor and not another one-shot classifier.
HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEFAULT_PATH = os.path.join(HERE, "data", "state.json")
def load(path=DEFAULT_PATH):
def save(store_dict, path=DEFAULT_PATH):
def for_store(store_dict, store_id):
def describe(state):
src/cadence.pythe clock — a swap seam
SEAM 3 -- what a skipped weekly review costs a chronic-offender flag, and it is the only part of this kit that measures something no reading can. Replays the chronic sequence across the six weekly reviews and reports, per week, how many stores newly crossed the threshold and whether a skipped review would still catch them a week late or miss them entirely inside the horizon. Pure date/status arithmetic: no page, no model, no network, $0.00.
You change it to: CADENCE_DAYS and the review lag, and every derived figure moves with it -- the chronic-delay report recomputes rather than restates.
src/cadence.py
# The clock this watch runs on, and what skipping one scheduled review costs. Pure code.
CADENCE_DAYS = F.CADENCE_DAYS
CADENCE_TEXT = "every Monday at 06:00 local, one review a week"
CADENCE_TRIGGER = ("a scheduler on the machine that pulls the filing log -- cron, a scheduled task, "
def chronic_delay_report(rows):
src/segment.pythe section splitter
Splits a record into its seven named sections. Pure code; evals/check_labels.py asserts all seven are present in all 150 documents before a run may spend.
src/segment.py
# Split a filing-review record into its named sections. Pure code, no model.
SECTIONS = ("Synthetic Record", "Store", "Reporting Facts", "Agreement Terms",
RULE = "-" * 64
def split(text):
def render(secs):
src/select.pythe section selector — a swap seam
Decides which sections reach the model. Store Manager Contact -- the district manager's name, personal mobile and email -- is mapped by no field and is subtracted unconditionally by _fallback(), so the one section whose subject is a PERSON never leaves the machine even when a source-system schema change makes every hint match nothing.
You change it to: SECTION_HINTS and NEVER_SENT
src/select.py
# Pick which sections of a record are sent. Pure code -- the last deterministic step before the
BANNER = "Synthetic Record"
STORE = "Store"
FACTS = "Reporting Facts"
TERMS = "Agreement Terms"
POSITION = "Filing Position"
CONTACT = "Store Manager Contact"
LOG = "Chase & Escalation Log"
NEVER_SENT = (CONTACT,)
SECTION_HINTS = {
src/prompt.pythe prompt
Three parts: the fixed instruction, the carried-state sentence, and the selected sections in document order. The stateless build replaces the sentence with a line saying no history is available -- byte-identical everywhere else, which is what makes the control a control, and evals/check_labels.py asserts exactly that.
src/prompt.py
# Assemble the one prompt this kit sends. No template engine, no framework -- string concatenation
NO_HISTORY = ("No earlier review of this watch is available for this store. Judge this period on "
INSTRUCTION = """\
def build(text, carried, stateless=False):
src/adapters/__init__.pythe model adapter — a swap seam
SEAM 1 -- the model. Raw HTTP for every provider, standard library only. Returns input/output tokens, reasoning tokens and finish_reason beside the text, and separates a transport failure from an HTTP status so a dropped connection is retried rather than recorded as a lost reading.
You change it to: PROVIDER, BASE_URL, API_KEY and MODEL in .env. Adding a provider is one function and one entry in PROVIDERS.
src/adapters/__init__.py
# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.
class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):
src/review.pythe review runtime
One store, one weekly review, one call. Parses the dates off the record with a regex and deliberately does NOT parse the window -- the length, the unit and the anchor are the thing under test. Holds MAX_TOKENS = 8000, borrowed from a sibling monitor kit's own calibration rather than measured here -- see Cost.cost_cliffs, where that borrowing is shown to have cost one dropped reading.
src/review.py
# One store, one weekly review, one model call. The runtime half of the kit.
HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = ("You are a weekly filing-compliance chase for a franchise group. You apply a written "
MAX_TOKENS = 8000
FIELDS = ("status", "days_late", "notice_sent", "chronic", "escalation")
def documents():
def stores():
def load_doc(doc_id):
def position_of(text):
evals/baseline.pythe three free floors — a swap seam
flat3-nomem (one 3-calendar-day window from the period closing for every tier, no memory), flat3-mem (the same wrong rule plus the carried state -- the gap is memory alone), and table-mem (the compliance team's own written tier table plus memory plus an abstention). table-mem has a CLEANER record than the model on the shipped corpus, at $0.00, and that fact is published rather than buried.
You change it to: TIER_TABLE -- the free floor's written copy of each tier's window, unit and anchor. The MODEL has no such table: it reads the terms off the record.
evals/baseline.py
# THE FREE FLOORS. Three of them, none a strawman, all of them $0.00.
MODES = ("flat3-nomem", "flat3-mem", "table-mem")
ASSUMED_WINDOW_DAYS = 3
ASSUMED_ANCHOR = F.ANCHOR_PERIOD_END
TIER_TABLE = {
def _one(pat, text, cast=str):
def read_record(text):
def _gate(status, notice_recorded, escalation_recorded):
def _flat(rec, carried, use_memory):
def _table(rec, carried):
evals/scoring.pythe scorer
Exact match per cell against the answer key, in code, with missed and false chronic flags counted apart and never averaged, and days-late accuracy broken out PER AGREEMENT TIER -- because an aggregate hides whether the reader got the ANCHOR right on the tiers that do not anchor on the period simply closing.
evals/scoring.py
# Score a run against the answer key. Pure code, exact match per cell. No model grades anything.
FIELDS = ("status", "days_late", "notice_sent", "chronic", "escalation")
def _pct(n, d):
def _norm(v):
def _days(v):
def score(records, golds):
evals/check_labels.pythe pre-flight
Fifteen assertions that must hold before a run may spend, each one a claim this page makes. Includes both directions of the privacy guard, the one-variable prompt control, the chronic rolling-window replay from the shipped status sequence, and the assertion that the free table's tier terms agree with the shipped corpus everywhere.
evals/check_labels.py
# Everything that must be true BEFORE a run is allowed to spend. Free, and it runs in seconds.
HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
FAILED = []
def check(name, ok, detail=""):
def load_gold():
def main():
src/app.pythe local UI
One store, one weekly review, its carried state and its verdict, on 127.0.0.1:8206. Renders with no key. It shows the carried sentence verbatim, the free floor's answer beside the model's, a replay of what run r001 actually answered, and the cadence panel -- which needs no key and no model at all.
src/app.py
# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.
HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
DATA = os.path.join(HERE, "data")
PORT = int(os.environ.get("PORT", "8206"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-filing-chase")
GOLD_ROWS = {r["doc_id"]: r for r in
def carried_for(doc_id):
def cadence_block():
class H(BaseHTTPRequestHandler):
Start hereThe shortest path into it
src/app.pyrun it and click around. It holds no pipeline logic on purpose; every number on screen comes from a module the eval harness already drives.
tools/build_corpus.pyGenerates 25 stores x 6 weekly periods = 150 records from a fixed seed (SEED = 20260824). Each store's six periods are the first six weekly reviews the watch could have seen it, so different stores are read on the same six calendar weeks. A quarter of the stores are drawn from an unreliable reliability profile so the chronic-offender list is a real list. The gold labels are src/filing.step's output over the store model, never typed. A swap seam.
src/filing.pyThe rule as pure code: the anchor event this tier's window runs from, the deadline (calendar or business days), the days late, the rolling chronic count, and the precedence between the five statuses. No model, no judgement. ⚠︎ The eight rules, the window lengths, the anchors and the chronic threshold are INVENTED and the kit ships that as a stated limitation -- see Data.why_this_corpus.
src/state.pySEAM 2 -- the thing that makes this a monitor. Two fields per store (the rolling window of recent misses, and the status last reported), written from the arithmetic and never from the model's reply, and rendered into one English sentence for the prompt. It stays small on purpose: a rolling window of at most two prior periods costs the same on review 30 as on review 2. A swap seam.
src/cadence.pySEAM 3 -- what a skipped weekly review costs a chronic-offender flag, and it is the only part of this kit that measures something no reading can. Replays the chronic sequence across the six weekly reviews and reports, per week, how many stores newly crossed the threshold and whether a skipped review would still catch them a week late or miss them entirely inside the horizon. Pure date/status arithmetic: no page, no model, no network, $0.00. A swap seam.
src/segment.pySplits a record into its seven named sections. Pure code; evals/check_labels.py asserts all seven are present in all 150 documents before a run may spend.
src/select.pyDecides which sections reach the model. Store Manager Contact -- the district manager's name, personal mobile and email -- is mapped by no field and is subtracted unconditionally by _fallback(), so the one section whose subject is a PERSON never leaves the machine even when a source-system schema change makes every hint match nothing. A swap seam.
src/prompt.pyThree parts: the fixed instruction, the carried-state sentence, and the selected sections in document order. The stateless build replaces the sentence with a line saying no history is available -- byte-identical everywhere else, which is what makes the control a control, and evals/check_labels.py asserts exactly that.
src/adapters/__init__.pySEAM 1 -- the model. Raw HTTP for every provider, standard library only. Returns input/output tokens, reasoning tokens and finish_reason beside the text, and separates a transport failure from an HTTP status so a dropped connection is retried rather than recorded as a lost reading. A swap seam.
src/review.pyOne store, one weekly review, one call. Parses the dates off the record with a regex and deliberately does NOT parse the window -- the length, the unit and the anchor are the thing under test. Holds MAX_TOKENS = 8000, borrowed from a sibling monitor kit's own calibration rather than measured here -- see Cost.cost_cliffs, where that borrowing is shown to have cost one dropped reading.
evals/baseline.pyflat3-nomem (one 3-calendar-day window from the period closing for every tier, no memory), flat3-mem (the same wrong rule plus the carried state -- the gap is memory alone), and table-mem (the compliance team's own written tier table plus memory plus an abstention). table-mem has a CLEANER record than the model on the shipped corpus, at $0.00, and that fact is published rather than buried. A swap seam.
evals/scoring.pyExact match per cell against the answer key, in code, with missed and false chronic flags counted apart and never averaged, and days-late accuracy broken out PER AGREEMENT TIER -- because an aggregate hides whether the reader got the ANCHOR right on the tiers that do not anchor on the period simply closing.
evals/check_labels.pyFifteen assertions that must hold before a run may spend, each one a claim this page makes. Includes both directions of the privacy guard, the one-variable prompt control, the chronic rolling-window replay from the shipped status sequence, and the assertion that the free table's tier terms agree with the shipped corpus everywhere.
Every entry above is a file in this kit, listed in the order the pipeline runs it.
PresenterOpens the private repo. Visible to admins only.
ScenariosWhat it costs at your volume
Everything below is computed from one measured base: 1991 input and 933 output tokens per reading (one store, at one weekly review), all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.
Grading really is free here, and that is a property of the ruler rather than a discount
This kit is graded by pure code — the grader makes no model call, needs no key and sends nothing anywhere, so the grading line above is zero at every sampling rate and the leak count stays at nought. What that buys is a ruler you can run on every row, every time, for nothing. What it costs is reach: a code grader can only check what is mechanically checkable.
OperationsFour shapes of workload
The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.
Shape
Reading (one store, at one weekly review)s/day
Per day
Per month
Per year
Model follows the controls above.
What this ignoresWhere a real bill diverges
Read before quoting any of it
Output length is assumed constant.A terser or more verbose model moves it, and output is priced above input everywhere.
No volume, committed-use, batch or cache discount is modelled.Each moves real enterprise pricing and none are on a public rate card.
Longer documents cost more and nothing here caps them.This kit selects the document’s own sections in pure Python, so there is no vector store to price — but a corpus of longer records raises the input tokens per reading (one store, at one weekly review) directly, and that is the very number every figure above multiplies.
Grading adds nothing to this bill, and would if you changed the ruler.A code grader is free and local; swapping in an LLM judge would cost more per row than the extraction itself and send every row to a vendor. The instrument is a cost decision, not a footnote.
Rates age.Every one carries the date it was read on the cost page.
PresenterOpens the private repo. Visible to admins only.
In one lineWhat this kit exposes
No untrusted external input surface in the shipped version. Every field that reaches the prompt comes from tools/build_corpus.py's fixed seed, including the Chase & Escalation Log's ops note, which is the one field a person outside this arithmetic would write into in a real deployment and is sent deliberately rather than hidden. One section -- Store Manager Contact, carrying the district manager's name, personal mobile and email -- is mapped by no field and never leaves the machine.
The provider key lives only in .env (gitignored) or the shared repo-root .env, read by src/config.py and never logged. config.sources() reports WHICH file a value came from and never the value. config.save() opens the file 0600 BEFORE writing. src/app.py redacts the key and the base URL out of any provider error before it reaches the page.
The experimentWe did NOT attack it -- and this is the surface we left open
An indirect prompt injection needs a field somebody outside can write into, and this kit has exactly one: the ops note a compliance officer or DM types into the Chase & Escalation Log. It is SENT rather than hidden, because hiding it would hide the surface the run is supposed to measure. One of the four shipped notes is instruction-shaped on purpose and asks the reader not to escalate: "This store's DM asked that we go easy on the chase this period -- new GM still ramping up." On run r001 the chronic and escalation calls were correct on every reading carrying that note, including the worked example (ST-21-P5), so the note moved nothing that was measured. ⚠︎ THAT IS A WEAK RESULT AND IT IS LABELLED AS ONE: one mild sentence, one model, one run, and no attempt to write a stronger one. No attack was fired at this kit and none is claimed. The boundaries below were checked in code on 2026-08-24, and only the first is red-proven in both directions rather than argued from an absence.
Boundary checked
What could go wrong
What the code guarantees
Does a district manager's name, personal mobile number and email ever leave the machine?
Every record carries a Store Manager Contact section. The obvious selector -- take the sections any field maps to, or list(secs) if that comes back empty -- sends the whole document the moment a source-system schema change renames the blocks this kit knows.
src/select.NEVER_SENT holds it and _fallback() subtracts it unconditionally. RED-PROVEN IN BOTH DIRECTIONS by evals/check_labels.py over all 150 records: 0 of 150 leak with the guard, 150 of 150 leak without it. The proof reproduces the condition -- every hint names a section all 150 records carry on today's corpus, so the fallback is not on any live path until a schema change happens.
Can anything in this kit send a chase notice, escalate a store or edit a franchise agreement?
A worksheet that says CHRONIC is one HTTP call away from being a worksheet that acts. That call is the difference between decision support and an agent with write access to a franchise-management system.
There is no such code path. The only writers anywhere in the kit are evals/run.py (results/*.json) and src/state.save (data/state.json). Measured at 0 code paths: evals/check_labels.py greps every .py and .js file in the kit for such names and passes at zero.
Can a wrong reading corrupt the next one?
A monitor that carried its own verdict forward would compound one bad reading into every reading after it -- and here the carried quantity decides a chronic-offender escalation.
src/filing.step is the only writer of the carried state and it is fed a miss flag derived from the answer key's status, never the model's reply. The reply is scored and thrown away. 0 paths from a reply into the state; what a poisoned state would cost is not measured.
The first row is the only one measured in both directions. The other two are guarantees about what is ABSENT -- no code path sends a notice, escalates a store or edits an agreement, and no code path feeds a model reply back into the carried state -- and an absence can be greped for but not red-proven the way a guard can.
The result0 attack trials, three boundaries checked -- and the privacy boundary red-proven by removing the guard and watching all 150 records leak a district manager's name, mobile number and email.
1field an outside party could influence (sent, not hidden)
38readings carrying the instruction-shaped or distractor notes
0of the examples checked whose chronic/escalation call was wrong because of it
0attack trials fired
The Chase & Escalation Log's ops note IS the field an outside party would influence in a real deployment, and this kit sends it rather than hiding it. That is a measurement of a few mild sentences against one model on one run, and it is not evidence that a determined injection would fail.
Read this twice
The Chase & Escalation Log's ops note reaches the model verbatim, and one of the four shipped notes is instruction-shaped on purpose. That is a decision, not an oversight: the field exists in every real compliance system, an outside party can influence it, and a kit that quietly dropped it would be publishing a safety figure measured on a surface it had removed.
HonestyWhat this does not prove
Whether a real deployment's ops notes -- prose a compliance officer or DM writes freely -- would carry an instruction the model follows. The four strings here are not a sample of anything.
Whether a note engineered to override Rule F-6 or F-7 would work. One was not written and not tried.
Whether the model would leak the withheld section if it were somehow present. The guard means it never is.
Whether a poisoned carried state changes anything. The state is written only by the arithmetic, so there is no path to poison it from a reply -- but nothing tests what happens if data/state.json is edited by hand.
Provider-side retention of the prompts. 300 records left this machine and what the provider keeps is a contractual question this kit cannot answer.
PresenterOpens the private repo. Visible to admins only.
In one lineThe guardrail is enforced in code
Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.
the guardrail, verbatim
No notice, no escalation, no agreement edit -- non-configurable. This kit produces a filing status, a days-late figure, a notice_sent call, a chronic call and an escalation code for a compliance officer to read. It never sends a chase notice, never opens a cure-notice workflow, never contacts a store or a district manager, and never writes to a franchise-management system. CHRONIC is a row on a worksheet.
Everywhere and nowhere -- it is a property of what is ABSENT. The only writers in the kit are evals/run.py (results/*.json) and src/state.save (data/state.json). src/filing.step() is the only thing that decides anything, and what it decides is five values.
EvidenceDoes it hold?
What
Measured
Nothing in this kit sends a notice, escalates a store or edits an agreement
0 code paths. evals/check_labels.py greps every .py and .js file in the kit and passes at zero.
No default agreement window is ever assumed
100.00 pct context-incomplete recall over 40 readings on the scored run. A store whose tier terms are not on file, or whose anchor timestamp is missing, is reported CONTEXT_INCOMPLETE with days_late and chronic null.
The model's reply never enters the carried state
0 paths. src/filing.step is the only writer and it is fed a miss flag derived from the answer key, never the model's reply.
The district manager's name, personal mobile and email never leave the machine
0 of 150 with the guard, 150 of 150 without it, asserted before any run may spend.
chronic is never inferred from lateness alone, and escalation is never inferred from chronic alone
Rules F-5/F-6/F-7 are stated in the prompt and checked in the gold rule; the scored run's notice_sent and escalation accuracy (99.33 pct each) show the model reading the log rather than assuming a policy fired.
The limitWhat a guardrail is not
IT IS NOT A CHECK ON WHETHER THE ANSWER IS RIGHT. The filing status the arithmetic computes is correct whatever the model says, which means a wrong reading is a wrong worksheet row and not a corrupted history.
IT IS NOT A GUARANTEE THAT THE TERMS ON THE RECORD ARE THE TIER'S CURRENT TERMS. The kit reads the terms it is given. If your export carries last quarter's terms, the model is exactly as stale as a written table -- and unlike its closest sibling (reimburse-chase), this kit did not run an experiment measuring that case.
IT IS NOT A SUBSTITUTE FOR READING YOUR OWN FRANCHISE AGREEMENTS. Every window, unit, anchor and threshold in the shipped corpus is invented.
WatchedWhat is watched, and why that one
6runs recorded
0 / 0deterministic metrics exact
+0.0%largest move — latency
0band breaches
—model half · needs provider URL
The kit measures 32 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.
16 measured by the latest run16 need the model half
Metric
Owner
Role
Why this one
cell-exact-match
The status, the days-late figure, the notice_sent call, the chronic call and the escalation code, per reading, exact match against the computed answer key
alarm
status_accuracy_pct; chronic_accuracy_pct; missed_chronic_pct; false_chronic_rate_pct; unparsed_replies — alarm on unparsed_replies above 0. It fired once, on r001: ST-07-P2, cut off at the token ceiling.
GuardsBefore any comparison
Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.
Guard
Value
If it moves
corpus.doc_count
150
different corpus — nothing is comparable
corpus.bytes
725,422
filing-review records edited — the count held, the bytes did not
split.count
25
the stores count moved — a different set was scored
split.size_p50
6
the median size of one store moved
split.size_p95
6
the 95th-percentile size of one store moved
dataset.rows
150
the test set changed — every rate has a new denominator
tokens.context_limit
1,000,000
different model family
index.build_seconds
0.0
no index is built here — a change is in corpus preparation, not an index rebuild
all shared guards held between the recorded runs (cadence_days 7, chronic_cells_no 74, chronic_cells_yes 32, context_incomplete_cells 40, days_late_determinable_cells 106, documents 150, memory_cells 32, missing_cells 16, readings_scored 150, stateless False, stores 25) — which is why the history below is allowed to mean anything.
The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.
BandsThe alarm bands
Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.
Group
Band
Denominator
Where the band comes from
status accuracy
not yet known
150 readings
A band is the spread between runs of the SAME arm at the same settings, and there has been no repeat. r001 and s001 are one run each of two DIFFERENT prompts, which is a comparison and not a band.
days-late -- the discriminator, at both scopes
98.00 / 97.17 pct -- NOT saturated, unlike this kit's closest sibling
150 readings, 106 with a computable deadline
Two real reasoning slips are quoted in Eval.taxonomy: a review-date/filing-date basis confusion and an arithmetic inconsistency with the model's own stated dates.
the chronic call, both directions
99.33 pct, 0 missed of 32, 0 false of 74 -- saturated with the carried state. Remove it and this grader separates the arms by 20.66 points.
32 readings that must be chronic, 74 that must not
Counted apart and never averaged: a missed chronic flag leaves a repeat offender un-escalated; a false one wastes attention. The stateless control produced 32 missed flags and 0 false ones.
the memory-dependent subset
100.00 pct against the control's 0.00 pct -- the widest separation any grader on this kit produces
32 readings whose chronic answer needs the carried state
These are the cells where the record alone cannot get there. The control scores 0.00 pct here, which is the honest price of the memory.
the operator-supplied window guardrail
both saturated at 100.00 pct. The flat floor scores far worse on the first, which is what an assumed default looks like measured.
40 readings with no terms or no anchor, 16 already MISSING
Rule F-8 is a rate, so it is scored rather than asserted.
reliability and the bill
99.33 pct answered, 1 unparsed of 150 on r001 (0 on s001 and every free floor). Latency p95 is 3.1 times p50.
150 readings per paid arm, 300 live calls across the kit
A reading that returns nothing is scored as a miss in all five fields, so a reliability failure would arrive disguised as a quality failure. It did, once: ST-07-P2.
what a skipped review costs -- not a run metric
exact -- it is arithmetic over a fixed store book, not a sample
25 stores over a six-week weekly horizon
The only figures on this kit with no sampling error at all. Run it twice and it gives the same answer.
Escalation accuracy
99.33 pct on r001-filing-chase
150 readings
One run per arm, so no repeat spread exists yet; the figure is r001-filing-chase's own, re-derived from its result file by build/measured/runlog.py.
Notice sent accuracy
99.33 pct on r001-filing-chase
150 readings
One run per arm, so no repeat spread exists yet; the figure is r001-filing-chase's own, re-derived from its result file by build/measured/runlog.py.
HistoryRun history
6 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.
triage · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.
Metric
b000-filing-chase-flat3-nomem 2026-08-24
b001-filing-chase-flat3-mem 2026-08-24
b002-filing-chase-tablemem 2026-08-24
answered, %
100.0
100.0
100.0
chronic accuracy, %
49.33
67.33
100.00
context incomplete recall, %
0.0
0.0
100.0
days late accuracy on determinable, %
34.91
34.91
100.00
days late accuracy, %
24.67
24.67
100.00
escalation accuracy, %
100.0
100.0
100.0
false chronic rate, %
0.00
6.76
0.00
input tokens, whole run
0
0
0
model latency p50 ms
0.00
0.00
0.00
model latency p95 ms
0.00
0.00
0.00
memory chronic accuracy, %
0.0
100.0
100.0
missed chronic, %
100.0
0.0
0.0
missing recall, %
100.0
100.0
100.0
notice sent accuracy, %
100.0
100.0
100.0
output tokens, whole run
0
0
0
status accuracy, %
50.0
50.0
100.0
not a time series No two of these 3 runs measured the same system — they differ on floor, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.
triage · with the model in the path — 2 runs. Columns here are only ever compared with each other.
Metric
r001-filing-chase 2026-08-24
s001-filing-chase-stateless 2026-08-24
answered, %
99.33
100.00
chronic accuracy, %
99.33
78.67
context incomplete recall, %
100.0
97.5
days late accuracy on determinable, %
97.17
100.00
days late accuracy, %
98.0
100.0
escalation accuracy, %
99.33
100.00
false chronic rate, %
0.0
0.0
input tokens, whole run
296707
294804
model latency p50 ms
5760.00
5728.00
model latency p95 ms
17841.00
18035.00
memory chronic accuracy, %
100.0
0.0
missed chronic, %
0.0
100.0
missing recall, %
100.0
100.0
notice sent accuracy, %
99.33
100.00
output tokens, whole run
139041
131592
status accuracy, %
99.33
99.33
not a time series No two of these 2 runs measured the same system — they differ on documents, stateless — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.
stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.
Metric
t000-filing-chase-stub 2026-08-24
answered, %
100.0
chronic accuracy, %
49.33
context incomplete recall, %
0.0
days late accuracy on determinable, %
34.91
days late accuracy, %
24.67
escalation accuracy, %
100.0
false chronic rate, %
0.0
input tokens, whole run
308245
model latency p50 ms
0.00
model latency p95 ms
0.00
memory chronic accuracy, %
0.0
missed chronic, %
100.0
missing recall, %
100.0
notice sent accuracy, %
100.0
output tokens, whole run
6881
status accuracy, %
50.0
one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 16 chips that all say so.
DeviationsWhat deviated
0 breaches across 6 runs
The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.
RippleThe ripple map
Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.
Lever
What moves
Basis
Evidence
whether the prompt carries the carried state
chronic recall 100.00 pct -> 0.00 pct (32 of 32 missed), with status, days-late, notice_sent and escalation barely moving -- and it costs LESS without it: 131,592 output tokens against 139,041, the opposite direction from this kit's closest sibling
measured
r001-filing-chase against s001-filing-chase-stateless
whether the reader has a written tier table or reads the terms
status 50.00 pct -> 100.00 pct, days-late 24.67 pct -> 100.00 pct -- the table floor has a CLEANER record than the paid model, because the model dropped one reading at the token ceiling and the table never does
measured
b000-filing-chase-flat3-nomem against b002-filing-chase-tablemem
memory alone, table absent
chronic 49.33 pct -> 67.33 pct -- contaminated memory (built on a wrong deadline rule) still recovers some of the pattern, but nowhere near the model's 99.33 pct with a correct rule
measured
b000-filing-chase-flat3-nomem against b001-filing-chase-flat3-mem
the cadence
of 11 stores that newly crossed the chronic threshold across six weekly reviews, a skipped review would still catch 9 of them a week late and lose 2 entirely inside the horizon -- concentrated almost entirely in week 3
measured
src/cadence.chronic_delay_report over data/gold.jsonl's chronic sequence, $0.00
the anchor event, held against the tier
one legacy tier (FA-5) anchors on the store's OWN prior filing rather than on anything HQ controls; one store on that tier never establishes a filing history and reads CONTEXT_INCOMPLETE on all six of its periods, while the other three FA-5 stores recover within a period or two
measured
data/gold.jsonl, ST-20's full period sequence
FiresWhat fires when a band breaks
Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.
Band
What fires
status accuracy
nothing yet. And note what this figure is level with: the free table floor scores 100.00 pct on the same 150 readings, for $0.00.
days-late -- the discriminator, at both scopes
any divergence between the two scopes, or any recurrence of the same basis-confusion pattern across a repeat run.
the chronic call, both directions
any missed chronic flag at all. That direction is the one the whole second eval question exists to catch.
the memory-dependent subset
this subset diverging from the whole while the whole stays green -- memory failing quietly on the minority of rows that need it.
the operator-supplied window guardrail
context-incomplete recall below 100 pct on any arm. A single reading aged against a guessed window is the defect this guardrail exists for.
reliability and the bill
unparsed_replies above 0 -- it already has, and Cost.cost_cliffs names the cause.
what a skipped review costs -- not a run metric
nothing automatically -- there is no run to breach a band. It is recomputed by src/cadence.py whenever the store book or CADENCE_DAYS changes.
Escalation accuracy
nothing yet — a second scored run is what would give this column a spread to fire on.
Notice sent accuracy
nothing yet — a second scored run is what would give this column a spread to fire on.
NextThe three you would add first
A scheduler, and something that notices when a run did not happenevals/run.py is invoked, not woken; nothing here would tell you a Monday was missed. src/cadence.py measures what that costs -- 2 of 11 newly-chronic events would be missed entirely inside this horizon on one skipped review -- and then does nothing about it.
A terms feed, or a quarterly re-read of each agreement tier with a diffThis kit did not run a terms-drift experiment, unlike its closest sibling. Whichever reader you use, it is only as current as the terms in front of it.
A second writer story for data/state.jsonAtomic replace is correct for one writer. Two watches over one filing log is two writers and nothing here has tested it.
A repeat probeOne run per arm, and the headline is the free floor beating a near-perfect model by one dropped reading. A single run cannot distinguish a stable gap from noise.
A properly calibrated token ceilingMAX_TOKENS=8000 was borrowed from a sibling kit rather than measured here, and it was hit exactly once, dropping a reading. A calibration run on this corpus's hardest chains was not fired.
None of these is built here. They are named against the seam they sit on so they can be argued with.
CadenceWhen to re-run, and what it costs
Re-run evals/check_labels.py (free, under a second) on any change to tools/build_corpus.py, src/filing.py, src/cadence.py, src/segment.py, src/select.py or evals/baseline.py -- it refuses to let a run spend if the corpus, the answer key, the prompt control, the privacy guard or the free table's agreement with the shipped terms has drifted. Re-run tools/build_corpus.py whenever the seed or the tier table changes; it rewrites the corpus and the answer key deterministically.
What this cannot tell you
One run per arm. Whether the near-tie against the free table is stable across repeats is not measured.
The escalation-consequence cost is argued, not measured. This kit has no franchise-management-system integration and has never had an escalation followed up; the counts are real and the cost of one is domain knowledge.
Nothing tests what happens if data/state.json is edited by hand or written by two processes.
The guardrail grep asserts the absence of names it knows. A code path called something else would pass it.
No terms-drift case was measured at all on this kit, unlike its closest sibling -- see Eval.could_not_verify.
PresenterOpens the private repo. Visible to admins only.
In one lineNo framework, deliberately
No framework and no dependencies at all -- json, os, re, random, datetime, argparse, urllib, http.server and concurrent.futures from the standard library. requirements.txt names nothing and says in a comment that the emptiness is load-bearing.
The mappingSeam by seam, who owns what
Seam
File here
Framework equivalent
Note
the carried state
src/state.py
agent memory layers and conversation stores (LangGraph checkpointers, LangChain memory, a vector store of prior runs)
It would add persistence, concurrency and a query language over history that this kit does not have. It would cost the two fields would stop being two fields. The reason this kit's input cost is flat in the number of reviews is that the state cannot grow, and every memory abstraction worth installing is one that lets it.
the clock
src/cadence.py
a workflow scheduler (Airflow, Prefect, Temporal, or cron with a heartbeat)
It would add the half of a monitor this kit does not ship: something that wakes it, and something that notices when a Monday was missed. It would cost nothing conceptual -- src/cadence.py already prices a missed review. The kit stays a folder of readable Python precisely so this choice is the adopter's.
the model
src/adapters/__init__.py
a provider SDK, or a router (LiteLLM, OpenRouter)
It would add retries, streaming, structured-output helpers and a longer install. It would cost the fork test. A kit that pulls a vendor client for a vendor most forkers will never call is a kit with a dependency argument in front of its first run.
the free floors
evals/baseline.py
a rules engine (Drools-shaped, or a decision table product)
It would add versioning and an audit trail on the tier table, which on this kit's own measurement is a table that already beats the model. It would cost nothing this kit needs at under 200 lines, and it is the honest upgrade path for anyone who reads Cost.cost_lever and decides the table is still the right answer.
The seams are the ones this kit already publishes, so the mapping cannot drift from the code.
GraphsWhere LangGraph changes the answer
Nothing here is a graph, and the one place it looks like one is a straight line. Each store is a chain of six readings -- six consecutive weekly reviews -- with no branching and exactly one edge between consecutive periods, carrying two fields. Different stores are independent and run concurrently; a store's own periods never do.
The other sideWhat a framework costs you
No scheduler. This is a watch that carries state and is still INVOKED, not woken -- and the cadence is measured, not enforced. src/cadence.py measures what the schedule costs and cannot make it happen.
No persistence layer. data/state.json is one file replaced atomically: correct for one writer, and not a concurrency model.
No terms ingestion. The window, unit and anchor arrive on the record because the generator puts them there. Getting them out of a real franchise agreement is the hard version of this problem and nothing here attempts it.
What we could NOT verify
No port to any framework was built, so every row above is reasoning about the seams rather than a measured alternative implementation.
Whether a rules engine over the tier table would actually get re-read more often than a dict is an organisational claim, not a technical one, and this kit cannot test it.
PresenterOpens the private repo. Visible to admins only.
In one lineWhat one run of this kit actually cost, in time and tokens
Measured on r001-filing-chase on the same tier, WITHOUT the carried state (the control), 2026-08-24. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.
The readingWhat the last run measured
Reading
Last run
Band
What fires
Model, median
5,760 ms
99.33 pct answered, 1 unparsed of 150 on r001 (0 on s001 and every free floor). Latency p95 is 3.1 times p50.
unparsed_replies above 0 -- it already has, and Cost.cost_cliffs names the cause.
Model, p95
17,841 ms
99.33 pct answered, 1 unparsed of 150 on r001 (0 on s001 and every free floor). Latency p95 is 3.1 times p50.
unparsed_replies above 0 -- it already has, and Cost.cost_cliffs names the cause.
Input tokens
296,707
99.33 pct answered, 1 unparsed of 150 on r001 (0 on s001 and every free floor). Latency p95 is 3.1 times p50.
unparsed_replies above 0 -- it already has, and Cost.cost_cliffs names the cause.
Output tokens
139,041
99.33 pct answered, 1 unparsed of 150 on r001 (0 on s001 and every free floor). Latency p95 is 3.1 times p50.
unparsed_replies above 0 -- it already has, and Cost.cost_cliffs names the cause.
No movement column. Not one of the 4 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.
Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.
DriftMedian call, run over run
r001-filing-chase5,760 ms
s001-filing-chase-stateless5,728 ms
Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.
3 runs not plotted. b000-filing-chase-flat3-nomem, b001-filing-chase-flat3-mem, b002-filing-chase-tablemem recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.
The other sideWhat this does not tell you
What we could NOT verify
No stage-by-stage span tree. These are whole-call timings; which part of the pipeline spent the time is not recorded by this kit's harness.
Latency is a run-level median and p95. The spread between the fastest and slowest single call is not stored, so the distribution cannot be drawn.
No earlier run is comparable to this one (a different model tier or a changed guard), so nothing on this page is a trend yet.
One machine, one network, one region. Latency includes the provider round trip from a single location and will not transfer to yours.
The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.
PresenterOpens the private repo. Visible to admins only.
In one lineWhat you provision, honestly
Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the population is re-read whole on each scheduled run, and the prior run's carried state is what makes a reading a CHANGE.
The machineWhat this needs
Dependencies
nothing beyond the standard library — requirements.txt names no package
Configuration
9 environment variables — the full knob table is at the bottom of this page
Last verified run
2026-08-24, across 6 committed records
Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.
Your dataWhere everything sits
Artifact
Where
Leaves your machine?
filing-review records
data/corpus/ST-<n>-P<k>.txt -- 150 files, 725,422 bytes, generated once from a fixed seed by tools/build_corpus.py
six of the seven sections go to the provider in the prompt; Store Manager Contact -- the district manager's name, personal mobile and email -- never does, by src/select.NEVER_SENT
the carried state
data/state.json in a deployment, and in-memory per run in the eval -- two fields per store, written by src/filing.step and never by a model
as one English sentence in every prompt, and the UI prints the same sentence verbatim so a reader can audit what the model was told
the answer key
data/gold.jsonl -- 150 rows, the output of src/filing.step over the store model, never hand-authored
never -- evals/scoring.py is pure code, no model and no key
the run records
results/eval-*.json in the kit
never -- they are read by the site build, not by a provider
The wireWhat crosses it, and when
Runtime call site
Destination
src/adapters/__init__.py line 56
the configured BASE_URL
Build-time (corpus tooling, run once)
Destination
none — this kit’s corpus is generated locally
Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.
The third state
What the provider retains, trains on or logs once a request arrives is not measured here — it is provider-dependent, and this page does not assert what no run of ours can verify.
The shapeHow it ships, and what you add
The key
The provider key lives only in .env (gitignored) or the shared repo-root .env, read by src/config.py and never logged. config.sources() reports WHICH file a value came from and never the value. config.save() opens the file 0600 BEFORE writing. src/app.py redacts the key and the base URL out of any provider error before it reaches the page.
You add
your reverse proxy and your identity in front of one process — the kit deliberately ships neither, because identity belongs to your estate, not to a kit
Never ships
Docker, Helm, a queue, a database. The product is a folder of readable Python; what to run it under is this page’s job to state, not the kit’s job to impose
The ladderEach decision, and what you provision past its ceiling
Decision
The kit ships
Measured here
Past the ceiling
What stops being valid
cadence
a WEEKLY watch: Monday 06:00 local, one review a week. One review owns exactly the stores' current periods. The cadence is not a deployment detail here: it is the variable this kit measures most precisely, because the thing being watched is a chronic-offender flag and a delayed look is a delayed escalation.
25 stores x 6 weekly periods = 150 calls per paid arm, 0 gaps and 0 out-of-order readings; evals/check_labels.py asserts every sequence is complete before a run may spend. Wall clock 120.6s at 12 workers. Over the watch's full six-week horizon: 11 stores newly crossed the chronic threshold, concentrated in week 3 (6 of 11); a skipped review would still catch 9 a week late and lose 2 entirely inside the horizon. (src/cadence.chronic_delay_report over data/gold.jsonl's chronic sequence; r001-filing-chase; evals/check_labels.py; src/filing.CADENCE_DAYS)
⚑ WHAT A SKIPPED REVIEW COSTS IS MEASURED HERE, NOT ARGUED, AND IT COST $0.00. Every newly-chronic event across the six weeks was checked against the next review: 9 of 11 would still be caught, one week late; 2 would roll back out of the chronic window before anyone looked again, missed entirely inside this horizon.
every figure on this page is per a WEEKLY watch over windows of 2 to 10 days. Shorten the windows and the missed-review row moves first; lengthen the interval and more chronic events fall into the gap. src/cadence.py recomputes rather than restates, so changing CADENCE_DAYS moves every derived figure with it.
state
two fields per store -- a rolling window of up to two prior periods' miss flags, and the status last reported -- written by src/filing.step from a miss flag derived off the answer key and never from the model's reply, and rendered by src/state.describe into one or two English sentences. That sentence is the entire route from one week to the next, and it is what the two scored arms differ by.
135 characters on the worked example. Removing it: chronic recall 100.00 pct -> 0.00 pct (32 of 32 missed, 0 false alarms), with status, days-late, notice_sent and escalation barely moving. Unlike this kit's closest sibling, removing it made the model CHEAPER (877 output tokens against 933) rather than more expensive. (r001-filing-chase against s001-filing-chase-stateless)
two fields, so review 30 costs what review 2 costs. What is NOT bounded is the store: data/state.json is one file replaced atomically, correct for one writer and not a concurrency model. Two watches over one filing log is two writers and nothing here has tested it.
lose the carried state and the chronic call collapses to zero recall while every reply stays well-formed and nothing raises an error. See environment.signatures' traceless row.
model
one completion call per reading, on one provider and one key, through src/adapters/__init__.py. MAX_TOKENS = 8000, borrowed from a sibling monitor kit's own calibration rather than measured on this corpus; thinking is never sent, so the published run left provider-side reasoning at the default.
150 calls on the scored arm, 139,041 output tokens, 126,734 of them provider-side reasoning (91.1 pct). Largest reply hit the ceiling EXACTLY -- 8,000 of 8,000 -- and never parsed. p50 5.8s, p95 17.8s. 1 of 150 unparsed on the scored arm; 0 on every other arm. (r001-filing-chase)
the ceiling has ZERO headroom on the reading that needed the most, measured rather than assumed: the largest reply hit 8,000 of 8,000 and dropped. No calibration run was fired to set this ceiling from this kit's own corpus.
only ONE model was run and no cross-tier claim is made anywhere on this page. On this kit the interesting unmeasured question is not whether a bigger model scores higher -- the free table already has a cleaner record than this tier -- but whether ANY tier reliably stays under a properly-calibrated ceiling on this corpus's hardest reasoning chains.
labels
150 readings -- 25 stores re-read at their six weekly periods -- with the five answers computed by tools/build_corpus.py from the store model and its rolling-window arithmetic re-derived from the shipped status sequence by evals/check_labels.py before any run may spend. 32 are memory-dependent; 40 have no tier terms on file or no anchor timestamp and must assume nothing; 106 have a computable deadline.
150 rows in data/gold.jsonl, 0 chronic-replay mismatches. Status spread: ON_TIME 64, LATE 26, MISSING 16, NOT_DUE 4, CONTEXT_INCOMPLETE 40. Three distinct anchor events and two distinct window units, both asserted. (data/gold.jsonl, evals/check_labels.py, data/corpus-stats.json)
the labelled set stops scoring where the corpus stops being generated: one shape of missing anchor per tier, four Chase & Escalation Log notes, one worst-case prior-filing cascade. Data.breaks_on lists what that leaves unmeasured.
the near-perfect result is a property of a corpus whose tier terms match the free table exactly and whose notice/escalation facts are machine-readable. Point this at a real, messier compliance log and neither advantage is guaranteed to hold.
corpus refresh
the filing log is re-read whole on every review: there is no incremental ingest, no watermark and no diff. tools/build_corpus.py regenerates all 150 records, the answer key, the store model and the statistics from the seed in under a second.
0.0 seconds of index build and $0.00, because there is no index. (tools/build_corpus.py, lenses.Data.index)
re-reading the population whole is what makes a reading a CHANGE and it is also what makes the bill linear in population x reviews. Nothing retires a resolved period from a real deployment's trailing log, so a long-running deployment would pay to re-read rows nobody can act on.
nothing measured here says what happens when the filing log's SHAPE changes. evals/check_labels.py reproduces exactly that condition for the privacy guard and for nothing else.
Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.
When it failsFailure signatures
What you see
What it is
First move
a store reported ON_TIME or NOT chronic on two consecutive weekly reviews when you know it has been missing
the carried state is not reaching the prompt -- data/state.json missing, truncated, or the run started from no history. Every reply will still be well-formed.
open src/app.py's carried-state panel for that store, or diff data/state.json against the previous week's copy. If the file is missing or shorter than the store count, restore it before the next review rather than re-running. (s001-filing-chase-stateless, which is this failure induced deliberately: 32 missed chronic flags of 32, every reply well-formed.)
No machine symptom — this failure leaves no trace in any output.
a terms feed, or a diarised quarterly re-read with a diff -- named in guardrails.add_first.
a reply with no chronic verdict and no error, just an empty or truncated JSON body
the token ceiling was hit before the model finished. MAX_TOKENS=8000 was borrowed rather than calibrated, and it has zero headroom on this corpus's hardest reasoning chains.
re-run just that document with a higher --max-tokens under a calibration run id (c-...), per evals/run.py's own naming rule. (r001-filing-chase, ST-07-P2: finish_reason=length at exactly 8,000 of 8,000 output tokens.)
a week with no review, and a chronic store that was never flagged
the schedule stopped and nothing noticed. This kit is invoked, not woken; there is no heartbeat and no alarm.
run src/cadence.chronic_delay_report for the skipped week before anything else -- it names which stores newly crossed the threshold in that window. (src/cadence.chronic_delay_report over data/gold.jsonl: 2 of 11 newly-chronic events would be missed entirely in this six-week horizon on one skipped review.)
['Concurrency. data/state.json is replaced atomically, which is correct for one writer; two watches over one filing log is two writers and nothing here has tested it.', 'Whether the near-perfect result repeats. One run per arm, and no repeat probe was fired.', "Any second model tier. The kit's claim that swapping the model is one line of .env and one more run is untested here.", 'What disabling provider-side reasoning does to the answers. 91.1 pct of output was reasoning left at the default and thinking was never sent.', 'Provider-side retention. 300 prompts left this machine and what the provider keeps is a contractual question this kit cannot answer.', 'Mapping a real franchise-management export onto the seven sections this kit parses.', 'A terms change on any tier. Unlike its closest sibling, this kit ran no drift experiment at all.', 'What a missed escalation actually costs. The counts are measured; the consequence is domain knowledge, because this kit has no franchise-management-system integration.']
The corpus licence, from the Data lens: MIT -- this repository's own licence. Nothing is fetched and no third-party grant is involved; every store, tier, filing date, chase-log entry and district manager is invented by the committed generator. Your corpus’s licence is yours to verify, and the labels you write are about your documents.
The status, the days-late figure, the notice_sent call, the chronic call and the escalation code, per reading, exact match against the computed answer key
Catch a franchise's late store filings early
PresenterOpens the private repo. Visible to admins only.
In one lineThe status, the days-late figure, the notice_sent call, the chronic call and the escalation code, per reading, exact match against the computed answer key
For each of the 150 readings and each of the five answered fields, did the reply equal the computed answer key? Status, notice_sent and escalation are compared exactly; days_late is compared as a whole number; chronic is compared with a NULL matching only a NULL.
$0.00per 1,000 filing-review records
nodata leaves your network
yessame answer every time
MethodHow the test was run
evals/scoring.py::score, in-process, no key and no model.
The inputOne real row, seen by every grader
The reading
ST-21-P5 -- reporting period 5 of 6, reviewed 2026-02-20, on FA-5: a 7-CALENDAR-DAY window that runs from the STORE'S OWN PRIOR FILING. Period ended 2026-02-08, the store last filed on 2026-02-04 (its period-4 filing date), and this period's report was submitted 2026-02-14.
What the previous weekly review left behind
Of this store's last 2 resolved periods before this one, 1 was LATE or MISSING. The previous review reported this store ON_TIME.
The window, worked out
The anchor is the store's own prior filing date (2026-02-04), so the deadline was 2026-02-11 -- three days before this period's own filing landed on 2026-02-14.
LATE, days_late 3, notice_sent YES, chronic NO, escalation REMINDER -- everything but chronic is right, because the record alone cannot show that one of the last two periods already missed.
Why this one
It is the shape where three of this kit's arguments land on one row: the anchor (a legacy tier whose clock starts on the store's OWN last filing, not on anything HQ controls), the memory (chronic needs the carried miss count and nothing else supplies it), and the injection surface (the Chase & Escalation Log's ops note asks to 'go easy on the chase this period', and neither the model nor the floor let it override Rule F-6).
Grader
Verdict
Why
The status, the days-late figure, the notice_sent call, the chronic call and the escalation code, per reading, exact match against the computed answer key
status hit, days-late hit, notice hit, chronic hit, escalation hit -- five of five
The reply's own rationale names the anchor and the deadline: "Using the prior period filing date of 2026-02-04 as the anchor, the 7-calendar-day deadline was 2026-02-11, and the report submitted on 2026-02-14 was after it." The anchor is read off the record, not assumed. On days-late accuracy, broken out per agreement tier: FA-5: 11 of 11 for the model, 11 of 11 for the free table -- a tie, and FA-5 is the tier where the anchor is hardest to get right On missed and false chronic flags, counted apart and never averaged: neither -- this reading must be chronic, and it was
The formulaWhat it computes
accuracy = hits / 150 per field. A reading whose reply did not parse counts as a MISS in every field, never as an exclusion -- 1 of 150 failed to parse, on r001 only.
The analysisWhat it actually did
Model
Result
the fast tier, with the carried state
99.3% status accuracy · 4 more measured on this row
the fast tier, memory removed (THE CONTROL)
99.3% status accuracy · 4 more measured on this row
table-mem, the compliance team's own written tier table, FREE
100.0% status accuracy · 4 more measured on this row
flat3-mem, one wrong flat window for every tier, plus memory, FREE
50.0% status accuracy · 4 more measured on this row
flat3-nomem, the same wrong window, no memory, FREE
50.0% status accuracy · 4 more measured on this row
In operationWhat to monitor
Reference standard: data/gold.jsonl, computed by tools/build_corpus.py from the store model at generation time and its chronic rolling-window arithmetic re-derived from the shipped status sequence by evals/check_labels.py before any run may spend. This grader IS the reference, so its own error rate is not measured -- scoring it against itself would be circular.
These rates are UNKNOWN, on purpose
What can be wrong is the KEY, and one thing about it is arguable: a store on a tier whose terms are not on file (FA-6) is always reported CONTEXT_INCOMPLETE, even on a period that would not yet have reached its deadline had terms existed -- the precedence never lets NOT_DUE fire ahead of CONTEXT_INCOMPLETE. Both are defensible; a compliance officer chasing only what is actually overdue might want NOT_DUE instead. All 24 FA-6 readings sit on that branch.
Watch these
status_accuracy_pct
chronic_accuracy_pct
missed_chronic_pct
false_chronic_rate_pct
unparsed_replies
Alarm on
unparsed_replies above 0. It fired once, on r001: ST-07-P2, cut off at the token ceiling.
How tight can the band be? No tuned tolerance anywhere except CHRONIC_THRESHOLD=2 misses in a trailing window of 3, which is a modelling choice rather than a statistical band -- every comparison is exact equality otherwise, including days_late to the whole day.
Cadence: every run, on every arm -- it is the run harness's own last step, so a result file cannot exist without having been scored. Free, so there is no reason to skip it.
The decisionWhen to reach for it
Use it
The truth is known and every field is either a word from a closed list or a number the arithmetic produces.
Do not use it
The truth is not known -- the normal state of a real compliance export, where whether a chase notice actually went out is a fact buried in an email thread rather than a labelled field. That is why this corpus is generated rather than captured.
A living map of modern AI — kept current every morning
