The business caseThe problem this solves

'Which promises are past their date' is a SELECT and nobody needs a language model for it -- the days past the date are printed on the page. The question a collections desk actually has to answer is the next one: what does somebody DO about this arrangement today. The ladder that answers it counts SCHEDULED OBSERVATIONS rather than days, so three of the terms it needs are not on the snapshot in front of you: how many consecutive weekly runs this promise has already been broken at, what was recommended at the last one, and how much had been credited then. A ledger that shows '28 days past' shows none of them. Somebody working a promise-to-pay queue by hand every Monday: opening each arrangement, working out from the collections notes whether the promise the ledger shows is still the promise the customer agreed to, remembering how many weeks in a row it has already been broken and what was done about it last week, spotting the part payment that landed on a Wednesday and appears in no dated entry, and only then deciding who picks up the phone.

The inputThe actual watch snapshots

The corpus is 150 watch snapshots, 0.70 MB (jsonl 1 · txt 150). It is generated because the alternative is somebody's real collections book, and a collections book is a list of named people at named companies who owe money. Generating it also buys the one thing a monitor most needs and cannot buy any other way: a CHAIN. Every arrangement is observed three times a week apart with events planted in the gaps, so the kit can ask 'what changed since the last reading' rather than 'what does this page say'. The nine planted patterns cover a promise kept, a promise part-paid then kept, a promise broken at three consecutive runs, a promise broken then renegotiated, a promise renegotiated then broken again, a dispute that caps the ladder, cash arriving in good faith between runs, and a promise not yet due -- and 14 observations carry a renegotiation or a dispute recorded only in a collector's sentence, because that is where the ledger and the human record actually disagree.

Synthetic Record
----------------------------------------------------------------
This file is generated for a public evaluation kit. Every customer, contact, invoice
number, amount and collector's note in it is invented, and the collections policy
reproduced below is illustrative: it is not any company's credit-management standard.

Arrangement
----------------------------------------------------------------
Arrangement                       : PTP-0001
Account                           : ACCT-4007  (Mid-Market, Team plan)
Customer                          : Northgate Lab
Scheduled run                     : 2026-08-03 07:00 UTC  (run 1 of the weekly watch)
Promise recorded on               : 2026-07-24
Promise generation                : 1
Promised amount                   : 14,247.35 USD
Promised by                       : 2026-08-06
Invoices covered                  : INV-20013, INV-20014

Collections Policy
----------------------------------------------------------------
  The promise-to-pay watch runs once a week, every Monday at 07:00 UTC, over every open
  arrangement in the book. One run is one observation of an arrangement. The rungs below count
  OBSERVATIONS, not days: an arrangement seen broken at three consecutive weekly runs is on rung 5
  whether that took 15 days or 25.

  1. Nothing is due and nothing has been missed                          -> NONE
  2. The promise falls due within 2 days of this run and less than
     half the promised amount has been credited against it               -> REMINDER
  3. The promise is broken -- the promised date has passed and the
     credited amount is short -- and this is the FIRST scheduled run at
     which it has been broken                                            -> COLLECTOR_CALL

The outcomeWhat a good result looks like

Per observation: the band the promise is in, the rung of the written ladder the desk should be on, how many consecutive scheduled runs the promise has been broken at, and what is holding the money -- plus a five-field carried state the next run is judged against, written by code from the parsed figures and never from the model's reply.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Your ladder is arithmetic on figures your ledger already prints — a rule engine, and no model at all
  Measured here: free rules carrying five scalars scored 100.0 / 100.0 / 100.0 on band, rung and count against the model's 99.33 / 94.0 / 99.33, for $0.00 and no key.
• Your collectors' notes carry facts your ledger does not — the model, on the prose fields only
  Rewrite the notes into unseen phrasings and the free keyword table falls to 16.67 pct on the driver and 83.33 pct on the band; the model on the identical set scored 76.67 pct and 100.0 pct.
• You want the model to write the state as well as read it — no -- keep the state in code
  The carried quantity here is a counter that only climbs. A monitor that fed its own verdict into next week's prompt would turn one bad week into a credit-hold review two weeks later, and nothing downstream would notice.

And where nothing here is good enough:

• You want to run the watch fortnightly to halve the bill — do not, or change the ladder first
  Measured for $0.00 by evals/cadence.py: skipping one weekly run leaves 9 surviving observations on a LOWER rung than they should be, never reaches 24 rungs at all, delays 10 by a run, loses 9 credit-hold recommendations, and makes 12 ledger events invisible to every later snapshot -- because the activity window is 3 days and the gap would be 14.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-23. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-23 — r001-ptp-watch. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with no key configured reproduces everything except the model's answers: python3 tools/build_corpus.py rebuilds all 150 snapshots and the answer key byte for byte, python3 -m evals.check_labels runs the whole pre-flight, both free floors score in under a second, python3 -m evals.cadence prices a missed run, and python3 -m src.app renders the UI with every panel filled in. What it cannot do is call a provider, and the button says so in a sentence rather than an error.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per observation: one HTTP request carrying the assembled prompt, and the reply parsed to a band, a rung, a count and a driver code. Splitting the snapshot into sections, dropping the one section no field asks for, parsing the figures and advancing the carried state all happen outside this clock and cost no network at all. p95 (50,865 ms) is 3.4x p50 (14,901 ms) -- output length here is driven by how many of the nine rules interact on a given arrangement, not by page length: every snapshot is within a few hundred bytes of every other and replies ran from 534 to 9770 tokens.

Current processWhat it replaces

Somebody working a promise-to-pay queue by hand every Monday: opening each arrangement, working out from the collections notes whether the promise the ledger shows is still the promise the customer agreed to, remembering how many weeks in a row it has already been broken and what was done about it last week, spotting the part payment that landed on a Wednesday and appears in no dated entry, and only then deciding who picks up the phone.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• 4 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one watch snapshot is

one snapshot per arrangement per scheduled run -- 50 arrangements x 3 weekly runs, never chunked. The whole open book is re-read on every scheduled run, so a 'split' here is the population divided by time rather than a document divided by length.

SetupWhat the setup figure measured

There is no index and nothing was built, so both figures are a MEASURED zero rather than an unmeasured one: no embedding call, no vector store, no chunk file, nothing on disk under data/index/. A monitor re-reads its whole population on every scheduled run and the finding lives in the difference between two readings, so there is nothing an index could be built over. Regenerating the corpus and the answer key takes under a second and needs no key -- that is corpus GENERATION, not index building, and it is counted in Data.cold_clone rather than here.

LicenceLicence

MIT, the same as the repository. Nothing in data/ is derived from anyone else's work, so there is nothing here to attribute.

Bring your ownBring your own watch snapshots

Replace data/ entirely. Point src/watch.py's CORPUS at your own snapshots in the same seven-section shape (Synthetic Record, Arrangement, Collections Policy, Cash Position, Activity Log, Account Contact, Collector Notes), put your own book's rules and cadence in src/promise.py, and hand-label a gold set. The scorer, the pre-flight, both free floors, the cadence measurement and the prose probe all work unchanged on any corpus in that shape.

⚠︎ And what stops being true when you do: The measured result does NOT travel with the code. This kit's answer key is COMPUTED by the same nine rules the kit publishes, from inputs the generator planted -- which is a luxury of controlling both the generator and the rules, and is why a free rule engine scores 100 pct on three of the four fields. On a real book the figures are parsed from documents somebody else wrote, the rules are your credit policy rather than these nine, and the key is hand-labelled with an error rate nothing here has measured. Bring the harness; re-measure everything.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — but they are the input total apportioned by each part’s share of the characters, not a per-part reading from the provider’s counter. The total is measured; the split is arithmetic, and a part carrying denser text really costs more of it than this says.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 1,795 tokens assembled, 1,113 are snapshots — 62% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

REPLAYED from the kit's own src/prompt.build() for PTP-0045-R3, not logged by the run. That is only trustworthy because the replayed section list matches what the run recorded (sections_used), and because the carried state it is built from is recomputed by the same src/promise.step the run used. The per-part token counts are apportioned by character share of the run's measured mean input (1794.94 tokens) -- the provider bills one number for the whole prompt and never breaks it down, so the split is arithmetic on a measured total rather than a second measurement.

Full promptThe prompt in full

You are running one scheduled weekly observation of one promise-to-pay arrangement on a software
company's collections book, and deciding what the collections desk should do about it TODAY.

The collections policy, its rungs and its cadence are reproduced in the snapshot below. Apply them
exactly as written.

The policy counts SCHEDULED RUNS, not days. How many consecutive runs this arrangement has already
been broken at, what rung was recommended at the previous run, and how much had been credited as at
that run are NOT on this snapshot. What is known about earlier runs is stated under "Carried state"
and is the only history available to you. Do not assume anything about earlier runs beyond it.

Two things to read carefully, because the ledger and the collector disagree on purpose:
 - The Activity Log covers only the three days up to and including this run, while the watch runs
   every seven. Cash that arrived earlier in the week is in the Cash Position total and in no dated
   entry anywhere on this page. Rule 8 asks whether cash MOVED since the previous run, which you can
   only answer from the credited figure in the carried state.
 - A renegotiation (rule 6) and an open dispute (rule 7) count when the Activity Log records them OR
   when the collector's note for this run records them. A customer ASKING for new terms is not a
   renegotiation; a customer who has agreed new terms that replace the promise above is.

Answer with a single JSON object and nothing else:

{"band": "STANDING|AT_RISK|BROKEN|KEPT|RENEGOTIATED",
 "escalation": "NONE|REMINDER|COLLECTOR_CALL|ACCOUNT_MANAGER|CREDIT_HOLD_REVIEW",
 "consecutive_broken_runs": <whole number of consecutive scheduled runs, INCLUDING this one, at
                             which this promise has been broken; 0 if it is not broken at this run>,
 "top_driver": "DISPUTE_OPEN|AWAITING_PO|BANK_IN_FLIGHT|PART_PAID_SHORT|RENEGOTIATION_REQUESTED|NO_REASON_RECORDED",
 "rationale": "one sentence, naming the rule you applied and the count you carried"}

"band" is the state of the promise at this run after rules 6 and 7 have been read. "escalation" is
the rung after the cap in rule 7 and the hold in rule 8 have been applied. "top_driver" is why the
money has not arrived, mapped to one of the six codes above; use NO_REASON_RECORDED when nothing on
the page gives a reason that maps to the other five, rather than choosing the nearest.

Carried state
----------------------------------------------------------------
The previous scheduled run was on 2026-08-10. It reported this arrangement BROKEN and recommended COLLECTOR_CALL. Counting that run, the promise has now been broken at 2 consecutive scheduled runs before this one. As at that run, 1,777.01 USD had been credited against the promise. The arrangement was then at generation 1.

Watch snapshot
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
This file is generated for a public evaluation kit. Every customer, contact, invoice
number, amount and collector's note in it is invented, and the collections policy
reproduced below is illustrative: it is not any company's credit-management standard.

Arrangement
----------------------------------------------------------------
Arrangement                       : PTP-0045
Account                           : ACCT-4315  (SMB, Team plan)
Customer                          : Denholm Cranes
Scheduled run                     : 2026-08-17 07:00 UTC  (run 3 of the weekly watch)
Promise recorded on               : 2026-07-10
Promise generation                : 1
Promised amount                   : 7,108.02 USD
Promised by                       : 2026-07-20
Invoices covered                  : INV-20585

Collections Policy
----------------------------------------------------------------
  The promise-to-pay watch runs once a week, every Monday at 07:00 UTC, over every open
  arrangement in the book. One run is one observation of an arrangement. The rungs below count
  OBSERVATIONS, not days: an arrangement seen broken at three consecutive weekly runs is on rung 5
  whether that took 15 days or 25.

  1. Nothing is due and nothing has been missed                          -> NONE
  2. The promise falls due within 2 days of this run and less than
     half the promised amount has been credited against it               -> REMINDER
  3. The promise is broken -- the promised date has passed and the
     credited amount is short -- and this is the FIRST scheduled run at
     which it has been broken                                            -> COLLECTOR_CALL
  4. Broken at the SECOND consecutive scheduled run                      -> ACCOUNT_MANAGER
  5. Broken at the THIRD consecutive scheduled run or later              -> CREDIT_HOLD_REVIEW
  6. A RENEGOTIATED promise starts again: the band is RENEGOTIATED, the rung returns to NONE and
     the consecutive count returns to zero at that run. A promise is renegotiated when the Activity
     Log records a RENEGOTIATED entry, OR when the collector's note for this run records the
     customer agreeing new terms that supersede the arrangement above. A note in which the customer
     merely ASKS for new terms is not a renegotiation.
  7. CAP: while a dispute is open against any invoice this promise covers, the rung is capped at
     COLLECTOR_CALL however many consecutive runs have passed. A dispute is open when the billing
     ledger's dispute flag is set, OR when the collector's note for this run records the customer
     disputing a covered invoice.
  8. HOLD: if cash was credited against this promise since the PREVIOUS scheduled run, AND that
     run recommended COLLECTOR_CALL or higher, the rung does not climb above the rung recommended
     at that run. The consecutive count still rises; the action does not. A first breach is not
     held by this rule, however much cash arrived.
  9. A broken promise of 25,000.00 USD or more climbs one rung early: ACCOUNT_MANAGER at the
     first broken run and CREDIT_HOLD_REVIEW at the second. Rules 7 and 8 are applied after it.
 10. The driver is the single thing most responsible for the money not having arrived at this run.
     NOTHING_OUTSTANDING is used when, and only when, the promise has been kept in full.
     TERMS_RENEGOTIATION covers both a renegotiation agreed and one the customer has only asked
     for; the band, not the driver, is what distinguishes them.

  This watch produces a recommendation for a collector to read. It suspends nothing, holds no
  orders, writes nothing off and sends nothing to a customer.

Cash Position
----------------------------------------------------------------
As at this scheduled run:
Promised amount                   : 7,108.02 USD
Credited against the promise      : 3,198.61 USD
Outstanding on the promise        : 3,909.41 USD
Days past the promised date       : 28
Dispute flag on the billing ledger: not set

Activity Log
----------------------------------------------------------------
Entries dated in the 3 days up to and including this run. Anything older is reflected
in the Cash Position above and appears nowhere else on this page.
  no entries in the 3 days up to this run

Collector Notes
----------------------------------------------------------------
  2026-08-17  "They paid what they could this month and went quiet on the balance."

Raw responseThe raw response

{"band": "STANDING", "escalation": "NONE", "consecutive_broken_runs": 0, "top_driver": "BANK_IN_FLIGHT", "rationale": "Rule 1 applies because the promised date is 2026-08-06, three days after this first weekly run, so nothing is due or missed; the consecutive broken count is 0, and the collector note that the bank shows payment sent but not yet arrived maps to BANK_IN_FLIGHT."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Decide what to do about a client's broken payment promise — 150 watch snapshots. One model answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

Three of the four answered fields are closed sets -- five bands, five rungs, seven driver codes -- and the fourth is a whole number of scheduled runs. A judge is for answers whose correctness is a matter of reading; these are matters of equality, and asking a model to grade COLLECTOR_CALL == COLLECTOR_CALL would add cost, variance and a second thing to be wrong.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid. ⚠︎ THE THREE CARDS WERE NOT ALL VERIFIED ON THE SAME DAY: each row carries its own rates_as_of read from build/facts/models.json (google/gemini-3-flash 2026-08-18, openai/gpt-5-6-luna 2026-08-18, anthropic/claude-fable-5 2026-08-23), and this block's date is the newest of them rather than a single date asserted over all three.

Rates checked 2026-08-23. The provider that actually ran all 389 calls is kept out of these tables per this estate's naming rule, so nothing here is what was paid. The real spend for this kit's whole build is recorded in the shared call ledger, not on this page.

Grading unitWhat the grading figure prices

The GRADER is free. evals/scoring.py makes no model call, and neither do either of the free floors it compares against, nor evals/cadence.py, nor the pre-flight. The unit that costs money is the OBSERVATION, and it is priced in the Cost lens; pricing the ruler in the units of the thing it measures is how a $0.00 grader gets reported as expensive.

The gradersOne way to grade, and why it is the only one

⚑ READ THIS BEFORE ANY OTHER NUMBER ON THIS PAGE. The free memory floor scores 100.0 pct on the band, 100.0 pct on the escalation rung and 100.0 pct on the consecutive count over all 150 observations. It is exact. It costs $0.00, it needs no key, it runs in under a second, and the model does not beat it -- 99.33 pct, 94.0 pct and 99.33 pct on the same three fields. On the corpus this kit ships, THE HONEST DEPLOYMENT OF THIS USE CASE IS A RULE ENGINE. That is not an accident of a weak model and this kit says why: given the figures printed on the page and five scalars carried from last week, the ladder is arithmetic. There is nothing in it for a language model to read. ⚠︎ AND THE 100 PCT IS PARTLY AN ARTEFACT OF THE FIXTURE, WHICH IS WHY THE PROSE PROBE EXISTS. The corpus draws every collector note from a fixed pool and the keyword table was written knowing what is in it -- a matcher tested on the sentences it was written for. Rewrite the notes on the 10 prose-dependent arrangements into phrasings it has never seen and the free floor falls to 83.33 pct on the band, 63.33 pct on the rung and 16.67 pct on the driver, and starts recommending credit-hold reviews it should not (4 of 30). The model was run on the identical rewritten set and scored 100.0 / 100.0 / 76.67. THAT comparison, not the headline one, is what a buyer should read. ⚑ AND THE TWO FLOORS FAIL IN OPPOSITE, READABLE WAYS. The memoryless one NEVER under-escalates -- 0 of 150 observations, zero rungs below the key -- it only climbs too far, and its 25 unsupported credit-hold recommendations land almost entirely on the three patterns memory exists for: goodfaith_hold (12), broken_climbing_stale (9), broken_then_renegotiated (4). The memory floor's only weakness is prose: on the rewritten notes it files 29 of 30 drivers under NO_REASON_RECORDED, which is a keyword table meeting sentences it was not written for and answering 'nobody wrote it down' to almost all of them.

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-23. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same observation, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

389 live calls are on the shared ledger for this kit and every one of them is behind a figure on this page. GRADING ITSELF COSTS NOTHING: evals/scoring.py makes no model call, and neither do the two free floors, the cadence measurement, the pre-flight or the prose probe's free arm. A forker re-running the eval pays the pipeline bill and no second bill on top of it. Figures are projected onto the Google Gemini 3 Flash card, not the real spend.

Cost driversWhat actually moves the bill

• Output tokens. 2335.4 per observation against 1794.94 in, so output is 88.6 pct of the projected bill on the shared card.
• The collections policy, reproduced on every snapshot. It is roughly 2774 characters of the 7421-character prompt -- about 37 pct of every call's input -- and one scheduled run sends it once per arrangement, so a book of 50 costs 50 copies of it a week, forever. That is the price of not asking a model to apply nine rules it cannot see, and it is paid on every observation rather than once.
• MEMORY, AND IT PUSHES THE BILL DOWN RATHER THAN UP. Removing the five-scalar paragraph cost $0.009383 an observation against $0.007904 with it -- 1.19x -- because output grew from 2335.4 tokens to 2837.97. On this shape a carried counter is not an overhead you pay for accuracy; it is the cheaper configuration AND the accurate one.
• The CADENCE, which is the multiplier nobody prices. One observation is one call; a weekly watch over a book of N arrangements is N calls a week forever, not N calls once. That is the cost shape of a monitor and it is the one a per-document price hides.
• Provider-side reasoning, left at the default. 95.9 pct of this run's output was reasoning tokens that never reach the parsed answer and still count against max_tokens.

Your volumeWhat it costs at your volume

Linear in arrangements and linear in runs, with no threshold anywhere in the range tested. Ten times the book is ten times the bill: $0.007904 an observation, so a book of 500 watched weekly is about $3.95 a week and 100,000 is about $790. ⚠︎ THE ARITHMETIC ARGUES AGAINST THE KIT AT EVERY SCALE, because the free rule engine on the row above costs $0.00 and was exact on three of the four fields.

Model choiceWhy this model

One tier was run, and the comparison this kit is built for is not between two models -- it is between a model and free code. Point .env at your own model and every arm re-runs on your numbers.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every stateful number on these pages: 149 observations, one completion call each, one tier. The stateless control, the prose probe and the cadence ablation are accounted for separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

15 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pythe corpus generator — a swap seam

Generates 50 arrangements x 3 consecutive WEEKLY scheduled runs = 150 snapshots from a fixed seed (SEED = 20260823) across three customer segments. It plants nine patterns, of which 14 observations carry a renegotiation or a dispute recorded ONLY in the collector's prose and 9 carry a payment that moved outside the snapshot's activity window. The gold labels are src/promise.replay's output, never typed.

You change it to: the whole of data/. Point src/watch.py's CORPUS at your own snapshots in the same seven-section shape and hand-label a gold set.

# Generate the corpus and the answer key. Deterministic from one seed; no network, no model.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
SEED = 20260823
RULE = "-" * 64
RUN_DATES = [dt.date(2026, 8, 3), dt.date(2026, 8, 10), dt.date(2026, 8, 17)]
CUSTOMERS = [
FIRST = ["Dana", "Marcus", "Priya", "Tomas", "Aoife", "Kenji", "Lena", "Ruben", "Sofia", "Idris",

src/promise.pythe collections ladder — a swap seam

The nine rules as pure code: band, rung, the consecutive-run counter, the dispute cap, the good-faith hold and the large-balance shift. No model, no judgement. ⚠︎ The cadence, every rung and every threshold in it are INVENTED and the kit ships that as a stated blocker -- see Data.why_this_corpus.

You change it to: RUN_INTERVAL_DAYS, the five rungs, the reminder lead, the half-the-amount test, LARGE_BALANCE_USD, the dispute cap and the good-faith hold. All in one file, all invented, all meant to be replaced by your own book's.

# The promise-to-pay policy, and the arithmetic it implies. Pure code, no model.

RUN_INTERVAL_DAYS = 7
RUN_TIME_UTC = "07:00"
RUN_DAY_NAME = "Monday"
ACTIVITY_WINDOW_DAYS = 3
STANDING = "STANDING"
AT_RISK = "AT_RISK"
BROKEN = "BROKEN"
KEPT = "KEPT"
RENEGOTIATED = "RENEGOTIATED"

src/promise.pythe cadence — a swap seam

RUN_INTERVAL_DAYS = 7, every Monday at 07:00 UTC, over the whole open book. It is not a setting hanging off the run -- it IS the run's trigger, and the rungs count observations rather than days, so a missed run cannot be reconstructed by arithmetic. evals/cadence.py prices one for $0.00.

You change it to: RUN_INTERVAL_DAYS, the five rungs, the reminder lead, the half-the-amount test, LARGE_BALANCE_USD, the dispute cap and the good-faith hold. All in one file, all invented, all meant to be replaced by your own book's.

# The promise-to-pay policy, and the arithmetic it implies. Pure code, no model.

RUN_INTERVAL_DAYS = 7
RUN_TIME_UTC = "07:00"
RUN_DAY_NAME = "Monday"
ACTIVITY_WINDOW_DAYS = 3
STANDING = "STANDING"
AT_RISK = "AT_RISK"
BROKEN = "BROKEN"
KEPT = "KEPT"
RENEGOTIATED = "RENEGOTIATED"

src/state.pythe carried state — a swap seam

SEAM 2 -- the thing that makes this a monitor. Five scalars per arrangement (the previous run's date, its band, its rung, the consecutive-broken count and the cash credited as at that run), written by the arithmetic and never from the model's reply, and rendered into one short paragraph for the prompt. It costs the same at run 40 as at run 3.

You change it to: which scalars cross the week. Five here; a book with an instalment plan would carry the plan's next due date as a sixth. Change describe() and the prompt changes with it -- there is no template engine in between.

# The carried state -- what makes this a monitor and not another one-shot classifier.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEFAULT_PATH = os.path.join(HERE, "data", "state.json")
def load(path=DEFAULT_PATH):
def save(store, path=DEFAULT_PATH):
def for_arrangement(store, arrangement_id):
def advance(store, arrangement_id, days_past, credited, promised, renegotiated, dispute_open,
def _usd(v):
def describe(state):

src/segment.pythe section splitter

Splits a snapshot into its seven named sections. Pure code; evals/check_labels.py asserts all seven are present in all 150 snapshots before a run may spend.

# Split a watch snapshot into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "Arrangement", "Collections Policy", "Cash Position",
RULE = "-" * 64
def split(text):
def render(secs):

src/select.pythe section selector — a swap seam

Decides which sections reach the model. Account Contact -- a named accounts-payable clerk, their work email, their direct line and a billing address -- is mapped by no field and is subtracted unconditionally by _fallback(), so the one section whose subject is a PERSON never leaves the machine even when a hint matches nothing.

You change it to: NEVER_SENT. One tuple decides which sections are subtracted before anything is sent, and _fallback() subtracts them unconditionally.

# Pick which sections of a snapshot are sent. Pure code -- the last deterministic step before the

BANNER = "Synthetic Record"
ARRANGEMENT = "Arrangement"
POLICY = "Collections Policy"
CASH = "Cash Position"
ACTIVITY = "Activity Log"
CONTACT = "Account Contact"
NOTES = "Collector Notes"
NEVER_SENT = (CONTACT,)
SECTION_HINTS = {

src/prompt.pythe prompt

Three parts: the fixed instruction, the carried-state paragraph, and the selected sections in document order. The stateless build replaces the paragraph with a line saying no history is available -- byte-identical everywhere else, which is what makes the control a control.

# Assemble the one prompt this kit sends. No template engine, no framework -- string concatenation

INSTRUCTION = """\
def build(text, carried, stateless=False):

src/adapters/__init__.pythe model adapter — a swap seam

SEAM 1 -- the model. Raw HTTP for every provider, standard library only. Returns input/output tokens, reasoning tokens and finish_reason beside the text, and separates a transport failure from an HTTP status so a dropped connection is retried rather than recorded as a lost observation.

You change it to: PROVIDER, BASE_URL, MODEL in .env. Nothing else. The kit-local .env overrides the shared repo-root one key by key, so pointing one kit at a different model is one line.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/watch.pythe watch

One arrangement, one scheduled run, one call. Parses the amounts, the dates and the ledger dispute flag off the page with regexes (the model is never asked to read a figure), assembles the prompt, calls the model, parses the reply. Holds MAX_TOKENS = 20000, the published ceiling.

# One arrangement, one scheduled run, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = ("You apply a written collections policy to one promise-to-pay arrangement at one "
MAX_TOKENS = 20000
FIELDS = ("band", "escalation", "consecutive_broken_runs", "top_driver")
def documents():
def arrangements():
def scheduled_runs():
def load_doc(doc_id):

src/app.pythe local UI

One arrangement, one scheduled run, its carried state and both free floors beside the model's answer, on 127.0.0.1:9004. Renders with no key. Where the model agrees with the free memory floor, the page says the model bought nothing on that row.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
PORT = int(os.environ.get("PORT", "9004"))
def _gold():
GOLD_ROWS = _gold()
def carried_for(doc_id):
class H(BaseHTTPRequestHandler):
def main():

evals/baseline.pythe free floors

TWO of them. onesnapshot is the ageing report a desk already runs: parsed figures, the ledger flag, an ordered keyword table over the note, and the consecutive count INFERRED from the days past the date. ledgermem is the same rules carrying the same five scalars the model is given. 0 calls, $0.00, scored through the identical scorer.

# The free, no-model floors. TWO of them, because one would have flattered the model.

DRIVER_KEYWORDS = [
ASKED_PAT = r"has asked to|they want to re-cut|nothing agreed|have not accepted|would come back"
AGREED_PAT = (r"agreed with their|new date agreed|settled on the call|we have re-cut|"
DISPUTE_NOTE_PAT = r"disputes? |contested|not paying|will not pay|query on|is wrong and"
def classify_driver(note, credited, promised):
def read_page(text):
def _answer(band, rung, n, driver, why):
def review_onesnapshot(text, _state=None):
def review_ledgermem(text, state):

evals/cadence.pythe cadence cost

Replays every arrangement against a schedule that skips one run and diffs the two answer keys. 0 calls. It is the only place in this kit that measures the CLOCK rather than the reading.

# WHAT A MISSED SCHEDULED RUN COSTS. Pure code over the answer key. Zero calls, zero dollars.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
OUT = os.path.join(HERE, "results", "cadence-cost.json")
def main():

evals/paraphrase.pythe prose probe

Rewrites the collector note on the 10 prose-dependent arrangements into phrasings the keyword table has never seen, so the free floor's score is tested off its own training set. Written in one pass and neither side edited afterwards.

# THE PROBE THAT MAKES THE FREE FLOOR'S SCORE HONEST.

PARA = {
NOTE_RE = re.compile(r"(Collector Notes\n-+\n  \S+  \")(.*)(\")", re.S)
def is_prose_dependent(gold_row):
def _key(gold_row):
def rewrite(text, gold_row, seq):

evals/scoring.pythe scorer

Exact match per cell against the computed gold, split six ways an average would hide: the four fields, the observations where a rung fires, the quiet ones, the counter-reset subset, the memory-dependent subset, and the credit-hold recommendations the policy does not support. No judge model.

# Score a run against the computed gold. Pure code, no model, no judge.

FIELDS = ("band", "escalation", "consecutive_broken_runs", "top_driver")
RUNGS = ("NONE", "REMINDER", "COLLECTOR_CALL", "ACCOUNT_MANAGER", "CREDIT_HOLD_REVIEW")
QUIET_RUNG = "NONE"
COSTLY_RUNG = "CREDIT_HOLD_REVIEW"
NO_REASON = "NO_REASON_RECORDED"
NOTHING = "NOTHING_OUTSTANDING"
def _pct(n, d):
def _norm(v):
def _as_int(v):

evals/check_labels.pythe pre-flight

Refuses to let a run spend until the parser, the schedule, the privacy guard, the stateless control's one-place difference, the answer key's provenance and the absent-guardrail grep all pass. Every one of them fails SILENTLY at run time.

# Pre-flight. Everything that must be true BEFORE a run is allowed to spend.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
FAILURES = []
def check(name, ok, detail=""):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 1794 input and 2335 output tokens per observation, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

No untrusted external input surface in the shipped version. Every field that reaches the prompt comes from tools/build_corpus.py's fixed seed, including the Collector Notes. src/select.py names the notes as the field an outside party WOULD influence in a real deployment -- a customer's own words, quoted by a collector -- and sends them deliberately, so the surface is visible rather than hidden. On this corpus nobody outside the repository wrote a byte of it.

The provider key lives only in .env (gitignored) or the shared repo-root .env, read by src/config.py and never logged. config.sources() reports WHICH file a value came from and never the value. config.save() opens the file 0600 before the write rather than chmod-ing it after, so the key never exists at the default umask even for an instant. src/app.py redacts the key and the base URL out of any provider error before it reaches the browser.

The experimentWe did not attack it -- and the boundary that matters most on this vertical was proven by trying to break it

An indirect prompt injection needs a field an outside party controls that reaches the prompt, and this corpus has none. So the three rows above are boundaries, not payloads. One of them is red-proven in both directions before every run: evals/check_labels.py strips the sections that Account Contact's siblings map to, checks that the fallback STILL subtracts the contact, and then checks that the same stripped document WOULD carry it if the subtraction were removed -- so the test is known to be able to fail. That second half is the point. A guard that has never been made to fail is a guard nobody has tested, and this estate has already paid once for a red-proof that measured zero because the path it exercised was unreachable. Confirmed by assertion and by reading the recorded run, not by an attack trial, on 2026-08-23 -- no attack run was fired; see Eval.redteam_page.

Three boundaries, all of them properties of what is ABSENT rather than of something that runs. That is the honest shape for a kit whose whole guarantee is 'it recommends and does not act', and it is also the shape that is easiest to lose in a refactor -- which is why two of the three are asserted by a gate that runs before any money is spent.

The result0 attack trials, three boundaries checked -- and the privacy boundary red-proven in both directions by evals/check_labels.py before any run may spend.

The Collector Notes ARE the field an outside party would influence in a real deployment, and this kit sends them rather than hiding them -- rules 6 and 7 both do their work there, so a version that withheld them would be measuring a different task. On this corpus they are drawn from a fixed pool by a seeded generator, so there is nothing adversarial in them to catch. What IS measured is the privacy boundary, in both directions, before every run.

HonestyWhat this does not prove

• No injection attempt was fired at the collector's note, which is this kit's own stated surface. Nothing here measures whether a customer's quoted sentence can move the band.
• The privacy guard is asserted over the corpus that ships. A snapshot in a different shape -- a contact block inside the Activity Log, say -- would not be caught by a section-name rule.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

No action of any kind, non-configurable. This kit produces a band, a rung RECOMMENDATION, a count and a driver code for a collector to read. It never suspends an account, places a credit hold, cancels a subscription, adjusts, writes off or emails anybody, and there is no setting that makes it. Separately: the carried state is written by code from the parsed figures and the previous state; the model's four answers are scored and are never written back into the history the next run is judged against.

Everywhere and nowhere -- it is a property of what is ABSENT. The only writers in the kit are evals/run.py (results/*.json) and src/state.save (data/state.json). src/state.advance() -> src/promise.step() is the only thing that touches the carried state, and evals/run.py calls it after every observation including one whose call FAILED.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• Not a credit-control system. It answers one question about one arrangement at one point on a clock and hands the answer to a person.
• Not a dunning engine. It sends nothing to a customer and there is no template, no address book and no outbound path anywhere in the kit.
• Not a collections workflow. It opens no case, assigns no owner, sets no follow-up date and writes to nothing but results/*.json and data/state.json.
• Not a decision. The top rung is a proposal to REVIEW whether a credit hold is warranted, and the kit's own scoreboard says free code makes fewer unsupported ones than the model does.
• Not a scheduler. It is invoked rather than woken, so nothing in it can notice that a scheduled run did not happen -- which evals/cadence.py prices at 9 lost credit-hold recommendations and 12 permanently hidden ledger events per missed weekly run.

WatchedWhat is watched, and why that one

The kit measures 41 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

14 measured by the latest run 27 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (stateless False) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

10 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

triage · with the model in the path — 9 runs. Columns here are only ever compared with each other.

not a time series No two of these 9 runs measured the same system — they differ on counter_reset_units, documents, driver_open_cells, escalations_that_fire, max_tokens, provider, stateless, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 14 chips that all say so.

DeviationsWhat deviated

0 breaches across 10 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The guardrail is checked by evals/check_labels.py, which runs before any spend, and by reading src/watch.py's module docstring, which states it where the calls are made rather than only on a page.

In one lineNo framework, deliberately

No framework and no dependencies at all -- json, os, re, random, datetime, argparse, urllib, http.server and concurrent.futures from the standard library. requirements.txt names nothing and says in a comment that the emptiness is load-bearing: a line there means something under src/ or evals/ imports it.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph, and the one place it looks like one is a straight line. Each arrangement is a chain of 3 observations -- run 1, run 2, run 3 -- with no branching and exactly one edge between consecutive runs, carrying five scalars. Different arrangements never touch. What IS a real structure is the other axis: one scheduled run fans out across the whole open book at once, which is a for-loop 50 wide, not an orchestration.

The other sideWhat a framework costs you

• No scheduler, and on a cadence kit that is the biggest thing missing. This monitor is INVOKED, not woken: the clock, the retry policy and the question of what to do when a run does not happen are all outside the kit. evals/cadence.py measures what a missed run costs ($0.00) and can do nothing to prevent one.
• No state store worth the name. data/state.json is one file replaced atomically; a framework would bring a checkpointer with a concurrency model, which this does not have -- and on a counter that only climbs that gap is worse than on a per-request kit.
• No observability beyond what evals/run.py prints and writes to results/. No tracing, no dashboard integration, no alert on a run that did not fire.
• No retry or backoff beyond src/adapters' own bounded retry, which covers a busy provider and a dropped connection and nothing else. It did not cover the one reply this run lost, which returned HTTP 200 with an empty body.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-ptp-watch on the same tier, memory removed (THE CONTROL), 2026-08-23. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 8 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

4 runs not plotted. a000-ptp-watch-paraphrase-free, a002f-ptp-watch-skiprun-free, b000-ptp-watch-onesnapshot, b001-ptp-watch-ledgermem recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the population is re-read whole on each scheduled run, and the prior run's carried state is what makes a reading a CHANGE.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT, the same as the repository. Nothing in data/ is derived from anyone else's work, so there is nothing here to attribute. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineCell-exact match against the computed key

whether each of the four answered fields equals the key for that observation

MethodHow the test was run

python3 -m evals.run --run-id <id> ... ; evals/scoring.py scores every arm

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, computed by src/promise.replay over the planted inputs -- never hand-authored.

How tight can the band be? The alarm band cannot be finer than one row of the negative class. There are 51 quiet observations, so one false alarm is 1.96 pct and no band tighter than that is measurable on this set.

Cadence: re-run on every corpus change and on every model change. It is free, so there is no reason to run it less often than that.

The decisionWhen to reach for it