The business caseThe problem this solves

'What did this property earn this quarter' is a SELECT and nobody needs a language model for it. The question a rights operation actually has is the next one: has the minimum guarantee been recouped, did it happen IN THIS period, and may the overage be invoiced yet. Recoupment is a running position and a royalty statement is a record of one period's flow, so the balance is on no page — and which guarantee is even operative, and which of the licensee's deductions may come off royalties first, are decided by a clause that was drafted by whichever firm papered that deal. A royalty analyst opening each statement, pulling the deal's recoupment clause out of the contract file, working out which guarantee that clause makes operative, deciding which of the licensee's deduction lines the clause actually permits, adding the result to a balance held in a spreadsheet, and then deciding whether the crossing is new.

The inputThe actual royalty statements

The corpus is 90 royalty statements, 0.24 MB (json 1 · jsonl 1 · txt 90). A RECOUPMENT WATCHLIST IS A LIST OF CONTRACTS. Not numbers: agreements — who licensed what, where, on what guarantee, and what that guarantee's own clause says about how it is recouped. There is no public corpus of that and there never will be one; an executed licence and the statements under it are the exact documents a royalty audit is fought over. So it is generated, it says so on the first line of every page, and every clause in it is illustrative drafting that reproduces no real agreement, no industry form and no firm's precedent.

Synthetic Record
----------------------------------------------------------------
This file is synthetic. It was generated for an open evaluation kit and describes no
real licensor, licensee, title, contract, statement or sum of money. Deal MG-0001, accounting
period P1.

Deal
----------------------------------------------------------------
Deal reference        : MG-0001
Licensee              : Northgate Home Media
Licensed property     : "The Salt Harvest"
Territory             : Benelux
Media                 : transactional VOD, subscription VOD
Contract year         : Contract Year 1 (2025-01-01 to 2025-12-31)
Accounting period     : quarter ended 2025-03-31
Royalty rate          : 26.3% of Licensee net receipts

Recoupment Clause (reproduced verbatim from the executed agreement)
----------------------------------------------------------------
4.1  Licensee shall pay to Licensor a minimum guarantee of USD 350,000.00 recoupable from
     royalties accruing over the whole Term. The guarantee is recoupable but not
     returnable.
4.2  Royalties are applied against the guarantee until it is recouped in full, after
     which royalties accruing are payable as overage.
4.3  All direct costs of exploitation properly evidenced by the Licensee are recoupable
     against royalties before application under 4.2.

Statement of Account for the Period
----------------------------------------------------------------
Minimum guarantee for the Term                        :     350,000.00 USD
Gross royalties earned this period                    :      78,827.00 USD
Deductions taken by Licensee this period:
    digital ingest and encoding fees                        3,050.00 USD
    origination and plate charges                           4,680.00 USD

The outcomeWhat a good result looks like

Per deal per accounting period: the three-way status (UNRECOUPED, CROSSED_THIS_CYCLE, ALREADY_IN_OVERAGE), whether the overage may be billed yet, the cumulative overage in whole dollars, and which clause feature governed the call.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• A book whose deals all use one house form, already configured into a royalty system with the guarantee, the pool and the reserve terms keyed in — your existing royalty system, at $0.00
  The free clause-rule floor scores 100 pct on status, invoiceability and governing clause over all 90 periods here, and a configured system has strictly more information than it does. There is nothing for a model to add.
• A portfolio assembled through acquisition — several hundred agreements, different firms' drafting, nothing configured, recoupment tracked in spreadsheets — read the clause, by whatever means
  The measured gap between reading the CLAUSE and reading the STATEMENT is 22.2 points of status accuracy, 8 missed crossings and 8 false raises per 90 periods. That gap is the whole business case, and it exists whether the reader is a rule engine or a model.
• Deciding whether a crossing is NEW, across a quarterly close — anything that carries a position between runs
  The memory-blind arm misses 19 of 25 crossings and scores 40.0 pct on the 60 cells whose answer is not on their own page. This is not a model question at all — it is whether the thing you run remembers.
• A single clause shape you already understand, and you want the overage figure to the dollar — the arithmetic, in code
  The clause-rule floor is exact on 83 of 90 overage figures and 0.99 pct out across the portfolio. The seven it misses are cost-category mappings, not arithmetic.
• Deciding whether to invoice a licensee — a person, holding the agreement
  This kit produces a worklist. It has no route to an invoice and no route to a counterparty, and evals/check_labels.py asserts both as properties of the source.

And where nothing here is good enough:

• A clause shape this kit has never seen — an escalating overage rate, a per-territory sub-guarantee, an instalment guarantee against delivery — neither this kit nor its floors, yet
  src/recoup.py has five shapes and no sixth. Both floors and the answer key would be confidently wrong rather than silent, because nothing here detects a clause feature it has no code for.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-23. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-23 — r001-mg-recoup. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with NO key configured reproduces the corpus byte-identically (python3 -m tools.build_corpus), passes all nine pre-flight assertions, runs both free floors and the stub over all 90 periods, and serves the UI with every panel populated. Nothing in that path touches the network.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per accounting period: one HTTP request carrying the assembled prompt, and the reply parsed to a status, an invoiceability, an overage and a clause. Measured over all 90 periods of r001-mg-recoup. ⚠︎ THE TAIL IS FIVE TIMES THE MEDIAN — 33.6s p50 against 168.4s p95, and the slowest single period took 263s — because 98.2 pct of every reply is provider-side reasoning and the hard clause shapes reason for much longer than the easy ones. A deployment sizing on the median will be wrong about its own close window.

Current processWhat it replaces

A royalty analyst opening each statement, pulling the deal's recoupment clause out of the contract file, working out which guarantee that clause makes operative, deciding which of the licensee's deduction lines the clause actually permits, adding the result to a balance held in a spreadsheet, and then deciding whether the crossing is new.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• 4 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one deals (3 accounting periods each) is

No split and no chunking. The unit is a DEAL — a run of three consecutive quarterly statements processed strictly in order, because period 3's prompt contains a recoupment balance produced by period 2. Each statement goes whole into one call.

SetupWhat the setup figure measured

There is no index and no retrieval step. tools/build_corpus.py writes 90 documents and the answer key from a fixed seed with no network and no model, in under a second. The population is re-read whole on every scheduled run — that is what a monitor is.

LicenceLicence

MIT — this repository's own licence. Nothing is fetched and no third-party grant is involved; every deal, clause, licensee, title, statement, cash flow and gold label is generated here.

Bring your ownBring your own royalty statements

Replace evals/baseline.py FIRST, and not as an option. This kit's headline is that a free rule engine wins, and that claim is only worth anything if the rule engine is YOURS — the incumbent you already run, with your clause terms configured as they actually are. Then drop your statements into data/corpus/ as <DEAL>-P<n>.txt (nothing in src/ parses one; the whole page goes into the prompt) and replace src/recoup.py with your own operative-balance rule.

⚠︎ And what stops being true when you do: Every figure here is a property of a three-period corpus of 30 invented deals in five clause shapes, measured once. The denominators are small — 25 gold crossings, 18 already-over cells, 7 reserve-blocked cells — and a single cell moves a percentage by more than a point on all three. Nothing here is a claim about a real royalty book.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 6,794 characters assembled, 3,198 are instructions — 47% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Replayed from the kit's own src/prompt.py::build on MG-0009-P2 over the SHIPPED corpus, with the carried position src/recoup.step produces for that cell — a cross-collateralized deal in its second period, which is the one prompt where the pool sentence and the apportionment sentence both appear. The system message is prepended, as the adapter sends it.

Full promptThe prompt in full

You apply a written recoupment clause to one licensing deal for one accounting period. You answer with one JSON object and no other text.

You are reviewing one licensing deal for one accounting period and deciding whether the licensor's
minimum guarantee has been recouped, whether it was recouped IN THIS PERIOD, and whether any overage
may be invoiced yet.

The recoupment clause that governs this deal is reproduced on the statement below. Apply it exactly
as written; do not apply a clause from a different deal and do not assume a market-standard term the
clause does not state. Where the clause and the licensee's own statement arithmetic disagree, the
clause governs.

Recoupment is a RUNNING POSITION and the statement is a record of ONE PERIOD. What has already been
applied against the guarantee is not on the statement; what is known about it is stated under
"Carried position" and is the only history available to you. Do not assume anything about earlier
periods beyond it.

How to work it out:
 - Start from the amount already applied, as given in the carried position.
 - Add what THIS period applies: gross royalties earned, LESS the reserve withheld this period, PLUS
   any reserve released this period, LESS only those deductions the clause permits to be recouped.
   A deduction the licensee has taken on the statement is not necessarily a deduction the clause
   allows to come off royalties before recoupment.
 - Compare the running total against the guarantee the clause makes operative for this period. Read
   the clause to decide WHICH guarantee that is and WHAT it is measured against.
 - The guarantee is recouped when the running total reaches or passes it.

Answer with a single JSON object and nothing else:

{"status": "UNRECOUPED|CROSSED_THIS_CYCLE|ALREADY_IN_OVERAGE",
 "invoiceable": "YES|NO_NOT_CROSSED|NO_RESERVE_HELD",
 "overage_usd": <whole USD by which the running total now exceeds the operative guarantee, 0 if none>,
 "basis": "MG_PER_TERM|CROSS_COLLATERALIZED|MG_PER_PERIOD|RESERVE_HELD|COSTS_LIMITED",
 "rationale": "one sentence naming the clause you applied and the running total you reached"}

"status" is CROSSED_THIS_CYCLE only when the guarantee was NOT recouped before this period and IS
recouped now. A deal that was already in overage at the previous period is ALREADY_IN_OVERAGE, not a
new crossing. "invoiceable" is NO_NOT_CROSSED whenever the guarantee is still unrecouped, and is
YES or NO_RESERVE_HELD only once it has been recouped. "overage_usd" is the cumulative excess of the
running total over the operative guarantee at this period end, not this period's earnings, and it is
0 whenever the status is UNRECOUPED. "basis" is the ONE clause feature that decides this deal's
recoupment: choose CROSS_COLLATERALIZED when the advance is recouped across more than one Licensed
Property, MG_PER_PERIOD when a separate guarantee applies to each Contract Year and does not carry
over, RESERVE_HELD when the clause makes overage payable only once a returns reserve has liquidated,
COSTS_LIMITED when the clause restricts which costs may be recouped, and MG_PER_TERM when a single
guarantee runs across the whole Term with none of those features.

Carried position
----------------------------------------------------------------
As at the previous accounting period, 330,027.00 USD had been applied to this Property, against an apportionment of 315,000.00 USD that the agreement treats as administrative. Across every Property sharing the same advance, 528,043.00 USD had been applied against an advance of 945,000.00 USD. The guarantee was NOT yet recouped at that point. No returns reserve was outstanding. It was reported UNRECOUPED.

Royalty statement
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
This file is synthetic. It was generated for an open evaluation kit and describes no
real licensor, licensee, title, contract, statement or sum of money. Deal MG-0009, accounting
period P2.

Deal
----------------------------------------------------------------
Deal reference        : MG-0009
Licensee              : Meridian Reel Company
Licensed property     : "Brackwater"
Territory             : Australia & New Zealand
Media                 : subscription VOD, ad-supported VOD
Contract year         : Contract Year 1 (2025-01-01 to 2025-12-31)
Accounting period     : quarter ended 2025-06-30
Royalty rate          : 23.8% of Licensee net receipts

Recoupment Clause (reproduced verbatim from the executed agreement)
----------------------------------------------------------------
4.1  Licensee shall pay to Licensor a single cross-collateralized minimum guarantee of
     USD 945,000.00 in respect of ALL Licensed Properties under this Agreement. The allocation
     of that guarantee shown on any statement is for the Licensee's internal reporting
     only and is not a separate guarantee.
4.2  Royalties accruing on any Licensed Property shall be applied against the guarantee
     in 4.1 until it is recouped IN FULL. No Licensed Property earns out separately, and
     no overage is payable in respect of any Property while the guarantee in 4.1 remains
     unrecouped in aggregate.
4.3  All direct costs of exploitation properly evidenced by the Licensee are recoupable
     against royalties before application under 4.2.

Statement of Account for the Period
----------------------------------------------------------------
Cross-collateralized advance (all Licensed Properties) :     945,000.00 USD
Advance allocated to this property (reporting only)   :     315,000.00 USD
Gross royalties earned this period                    :     309,350.00 USD
Deductions taken by Licensee this period:
    origination and plate charges                           5,120.00 USD
    palletised carriage to retail depots                    2,960.00 USD
Returns reserve withheld this period                  :           0.00 USD
Returns reserve released this period                  :           0.00 USD
Applied this period by other Licensed Properties under the
same cross-collateralized advance                     :     180,762.00 USD

Licensee Contact
----------------------------------------------------------------
Royalty contact  : Meridian Reel Company
Contact email    : royalties@meridian.example.invalid
Contact number   : +1 555 0209

Licensee Remarks
----------------------------------------------------------------
Statement prepared on the licensee's standard reporting template. Figures are net of returns actually processed in the period and exclude any provision not shown above.

Statement Basis
----------------------------------------------------------------
This statement reports the ACTIVITY OF THIS PERIOD ONLY. It does not state the
recoupment position brought forward, and no figure on it is cumulative.

Raw responseThe raw response

{"status":"UNRECOUPED","invoiceable":"NO_NOT_CROSSED","overage_usd":0,"basis":"COSTS_LIMITED","rationale":"Applying clause 4.1 and 4.3, gross royalties of $78,827 less permitted direct exploitation costs of $7,730 gives $71,097 applied against the $350,000 guarantee, leaving it unrecouped."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check whether a licensing deal has earned out its guarantee — 90 royalty statements. Five tiers of one model family answered, and every answer was then graded Five different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

There is no LLM-as-judge in this kit. Three of the four answered fields are words from a closed list and the fourth is a dollar figure, so the grader is an equality test and a one-dollar tolerance. Grading calls nothing and costs nothing — a measured $0.00, not an unpriced one.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that ran the nine live calls is kept out of these tables per this estate's naming rule, so nothing here is what was paid. The real spend for this kit's whole build is nine calls on the shared ledger.

Grading unitWhat the grading figure prices

evals/scoring.py makes no model call, and neither do either of the free floors. The whole 90-cell scoring pass runs in under a second with no key and no network.

The gradersFive ways to grade

⚑ THE MODEL MATCHED THE FREE FLOOR AND BEAT NOTHING. r001-mg-recoup answered all 90 deal-periods: status 100.0 pct, 0 of 25 crossings missed, 0 false raises, invoiceability 100.0 pct, memory-dependent status 100.0 pct, 0 replies lost. Every one of those figures is EQUAL to what evals/baseline.py's clause-rule floor produces for $0.00, and on the two graders where they differ THE FREE FLOOR WINS: it names the governing clause on 100.0 pct of periods against the model's 68.89, and it is exact on the overage figure on 83 of 90 against the model's 81 (portfolio error 0.99 pct against 1.54). SO THE CONCLUSION IS UNCHANGED BY THE RUN, WHICH IS THE POINT OF HAVING RUN IT: on a portfolio whose clause shapes are configured, this job does not need a model and you should not pay for one. The model did not fail — it did the job correctly — it simply did not do anything the free reader was not already doing, and it charged $0.017491 a period to do it. THE GAP BETWEEN THE TWO FREE FLOORS IS STILL THE LARGEST MEASURED EFFECT ON THIS PAGE. The plain ledger — the same running balance, booking the licensee's own deduction column as presented, which is what a royalty system holds when the deals were keyed in from a deal memo — scores 77.78 pct and gets 8 of 25 crossings wrong and raises 8 false ones. Reading the CLAUSE rather than the STATEMENT is worth 22.2 points, and it costs nothing either way. Nothing the model did came close to that margin, because there was no margin left to take. AND THE MEMORY IS WHERE THE MODEL ARM EARNS ITS PLACE ON THE PAGE — not against the floor, against ITSELF. s001-mg-recoup-stateless is the identical prompt with the carried-position paragraph replaced by a sentence saying no history is available, and it scores 60.0 pct on status against 100.0, missing 18 of 25 crossings and 40.0 pct on the 60 memory-dependent cells. That is the one thing this kit measured that a free rule engine could not have told anybody: the position is load-bearing, and a per-statement classifier of any quality cannot do this job.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same deal-period, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

189 live calls is this kit's entire spend: 9 calibration, 90 scored, 90 control. Every one is behind a figure on this page. A further 90 were attempted earlier in the day and refused by the provider for lack of balance — those produced nothing and cost nothing, and no result file was kept, because a run named r001 holding ninety zeroes is how a zero gets read as a score. discarded_calls is the one calibration period billed for a reply that hit the ceiling and returned nothing parseable.

Cost driversWhat actually moves the bill

• PROVIDER-SIDE REASONING, 98.2 pct of output tokens on the scored run. It was left at the provider's default and this kit never sends a thinking field, so every figure here is measured with reasoning ON. src/adapters.THINKING_OFF exists and has never been sent.
• OUTPUT, NOT INPUT. The prompt is about 1468 tokens and the reply averages 5585.67 — roughly four output tokens for every input token, which inverts the usual shape and means shortening the statement saves almost nothing.
• ONE CALL PER DEAL PER PERIOD, flat in history length. The carried position is six scalars, so period 40 costs exactly what period 2 costs. That is the one place this shape is cheaper than a conversation.
• THE CADENCE, WHICH IS NOT A KNOB THIS KIT OWNS. The billing unit is a deal-period and the period is fixed by the agreements — quarterly, here. Running more often does not produce more answers, it produces the same answer again against a statement that has not changed.

Your volumeWhat it costs at your volume

Linear in DEAL-PERIODS and flat in history length. Ten times the book is ten times the calls at the same per-call cost, because nothing about period 40 is larger than period 2. The thing that does not scale linearly is WALL CLOCK: each deal's periods are strictly serial, so a quarter's close is as wide as your concurrency allows and as deep as three. At the measured 33.6s p50 and 168.4s p95 over 90 periods, a book of 8,000 open deals is 8,000 calls and roughly 75 hours of single-threaded provider time — about 7.5 hours at ten concurrent chains, which is a batch job and not an interactive one. ⚠︎ EXTRAPOLATED FROM 30 CHAINS; nothing here has run more.

Model choiceWhy this model

It is the tier the shared key is pointed at, and it is the cheap one — the right place to start a question whose answer might be 'you do not need a model for this'. On this kit that IS the answer: the cheap tier matched a free rule engine exactly, so a dearer tier could only have matched it more expensively. Nothing here ranks two models, and after this result there is no reason to buy a second one.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

All 90 deal-periods at the published 32,000-token ceiling, 0 replies lost. The control arm cost a further 128004 in and 510168 out over the same 90 periods.

The codeEvery file, and what it is for

12 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pythe corpus generator

Generates 30 licensing deals x 3 consecutive quarterly royalty statements = 90 documents from a fixed seed (SEED = 20260823), across five clause shapes and two drafting registers, and computes every gold label by calling src/recoup.replay over the planted cash flows.

# Generate the corpus: 30 licensing deals x 3 accounting periods = 90 royalty statements, plus

SEED = 20260823
HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
PERIODS = ("P1", "P2", "P3")
PERIOD_END = {"P1": "2025-03-31", "P2": "2025-06-30", "P3": "2025-09-30"}
PERIOD_SHAPES = {"P1": "Contract Year 1", "P2": "Contract Year 1", "P3": "Contract Year 1"}
PERIOD_YEAR_PERIODIC = {"P1": "Contract Year 1", "P2": "Contract Year 1", "P3": "Contract Year 2"}

src/recoup.pythe recoupment arithmetic — a swap seam

Integer-cent arithmetic: what one period applies, which balance the clause makes operative, whether the crossing is new, and what overage stands. It is the answer key and it is NEVER called at run time — src/monitor.py imports only the status codes from it.

You change it to: Your clause vocabulary and your operative-balance rule. The five shapes shipped here are five, not all: escalating overage rates, per-territory sub-guarantees, instalment guarantees and end-of-year shortfall top-ups are all ordinary and all absent.

# The recoupment arithmetic, and nothing else. Integers of cents, no dates, no model.

UNRECOUPED = "UNRECOUPED"
CROSSED_THIS_CYCLE = "CROSSED_THIS_CYCLE"
ALREADY_IN_OVERAGE = "ALREADY_IN_OVERAGE"
STATUSES = (UNRECOUPED, CROSSED_THIS_CYCLE, ALREADY_IN_OVERAGE)
INV_YES = "YES"
INV_NOT_CROSSED = "NO_NOT_CROSSED"
INV_RESERVE_HELD = "NO_RESERVE_HELD"
INVOICEABLE = (INV_YES, INV_NOT_CROSSED, INV_RESERVE_HELD)
B_TERM = "MG_PER_TERM"

src/state.pythe carried position — a swap seam

Six scalar fields between periods: applied, operative guarantee, its period label, whether an earlier period crossed, reserve still held, and the pool pair. Written from the arithmetic, never from the model's reply, and rendered into one English paragraph for the prompt.

You change it to: What persists between periods, and how it is worded for the prompt. Six fields here; a book with sub-guarantees needs more.

# The carried state -- what makes this a monitor instead of a very expensive calculator.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEFAULT_PATH = os.path.join(HERE, "data", "state.json")
def load(path=DEFAULT_PATH):
def save(store, path=DEFAULT_PATH):
def for_deal(store, deal_id):
def describe(state, shape):

src/prompt.pythe prompt builder

Three parts, concatenated: the instruction and JSON shape, the carried position, and the statement whole. The clause is NOT in here — it travels with the document, which is what makes this a test of applying a rule rather than of remembering one.

# Assemble the one prompt this kit sends. String concatenation you can read -- no template engine,

INSTRUCTION = """\
def build(text, carried, shape, stateless=False):

src/monitor.pythe period reviewer

One deal, one period, one completion call. Holds MAX_TOKENS, the published ceiling, and the tolerant JSON parse. Issues nothing and writes nothing.

# One deal, one accounting period, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = ("You apply a written recoupment clause to one licensing deal for one accounting period. "
MAX_TOKENS = 32000
FIELDS = ("status", "invoiceable", "overage_usd", "basis")
def documents():
def deals():
def load_doc(doc_id):
def _parse(txt):

src/adapters/__init__.pythe provider adapter — a swap seam

Raw HTTP to any OpenAI-compatible endpoint or to Anthropic's Messages API, stdlib only. Transient-vs-terminal retry, a transport-failure branch, and the daily call cap checked once per completion.

You change it to: .env plus the same run again. Any OpenAI-compatible provider or Anthropic.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/budget.pythe call cap

An append-only ledger beside the shared .env, written BEFORE each call, so a crash over-counts rather than under-counts. Shared by every kit under the repo root, because they share one key.

# A cap on live calls, shared by every kit on this machine. No dependency, no service.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
KIT = os.path.basename(HERE)
ROOT = os.path.dirname(os.path.dirname(HERE))
SHARED_ENV = os.path.join(ROOT, ".env")
LEDGER = os.path.join(ROOT if os.path.exists(SHARED_ENV) else HERE, ".calls-ledger.jsonl")
PAID_ARMS = ("r", "x")
FREE_ARMS = ("t", "b")
BOOT = os.urandom(6).hex()
PID = os.getpid()

evals/baseline.pythe two free floors — a swap seam

A cumulative ledger and a clause-rule engine, both pure Python, both reading the same statements the model reads, both keeping their own carried position. They open no file at all — check_labels asserts it.

You change it to: The comparison this whole kit is measured against. If your incumbent is a configured royalty system rather than a spreadsheet, this is the file that has to say so.

# The free, no-model floors. Two of them, and the second one is the hard bar.

RE_POOL = re.compile(r"cross-collateralized|single advance made in respect of the Properties|"
RE_PERIODIC = re.compile(r"NON-CUMULATIVE|each Contract Year thereafter|"
RE_RESERVE_BAR = re.compile(r"NO OVERAGE SHALL BE PAYABLE while any\s+returns reserve|"
RE_COSTS_LIMITED = re.compile(r"Recoupable costs are LIMITED to|"
PERMITTED_KEYWORDS = ("manufactur", "master", "encode", "encoding", "freight", "carriage",
MONEY = r"([\d,]+\.\d\d) USD"
P_GUARANTEE_TERM = r"^(?:Minimum guarantee for the Term|Guarantee for the Term)\s+: \s*" + MONEY
P_GUARANTEE_YEAR = r"^Guarantee for Contract Year \d\s+: \s*" + MONEY
P_POOL_ADVANCE = (r"^(?:Cross-collateralized advance \(all Licensed Properties\)|"

evals/scoring.pythe scorer

Six graders over 90 cells, with missed crossings and false raises counted against different denominators and never averaged, plus a per-clause-shape table and a status confusion matrix.

# Score one run against the answer key. Pure code, no model, no judge.

STATUSES = R.STATUSES
INVOICEABLE = R.INVOICEABLE
BASES = R.BASES
SHAPES = ("TERM", "POOL", "PERIOD", "RESERVE", "COSTS")
OVERAGE_TOLERANCE_USD = 1.0
def _pct(n, d):
def _overage_usd(ans):
def score(records, golds):

evals/check_labels.pythe pre-flight

Nine assertions that must pass before any run may spend: the sequence is complete, the gold re-derives, no statement names another period or prints a cumulative figure, the two prompt builds differ in one place, the floors reach no filesystem, the instruction carries no clause signal, and no module has a route to an invoice or to a licensee.

# Everything that must be true before a run may spend. No key, no network, no model.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
PERIODS = ("P1", "P2", "P3")
def rows():
def main():

evals/run.pythe run harness

30 deal chains, each strictly serial across its three periods, run concurrently against each other. Prints the spend before it spends.

# Run the watch over the 30 deals and write one result file. THIS SPENDS MONEY.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
ABLATED_RULES = [
def load_gold():
def stub_complete(cfg, system, user, max_tokens=1024):
def main():
def _facts(g):

src/app.pythe local UI

http.server on port 9002. Renders with no key: the statement, the carried position and both free floors are computed locally.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
PORT = int(os.environ.get("PORT", "9002"))
def _gold():
GOLD_ROWS = _gold()
def _facts(g):
def carried_for(doc_id):
def floors_for(doc_id):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 1468 input and 5585 output tokens per deal-period, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

No untrusted external input surface in the shipped version. Every field that reaches the prompt comes from tools/build_corpus.py, generated from a fixed seed. In a real deployment the untrusted surface is the LICENSEE REMARKS section — free prose the counterparty writes and the licensor reproduces — and on this corpus it is one of three fixed sentences.

The provider key lives only in .env (gitignored) or the shared repo-root .env, read by src/config.py and merged key by key. It is never logged, never written to results/, and the UI's error path rewrites it out of any message before returning it. src/config.sources() reports WHICH file contributed and never a value.

The experimentWe did not attack it — and the boundary that matters was proven by removing the guard, not by arguing about it

An indirect prompt injection needs a field an outside party controls that reaches the prompt, and on this corpus there is none. What WAS proven is narrower and checkable: swap evals/baseline.py's floors for versions that read data/gold.jsonl and both score 100 pct on everything, which is why check_labels asserts the file cannot open anything at all. The first version of that assertion banned the substring 'gold' and convicted the file on its own docstring — a checker that cannot tell a claim from a route is a checker that gets switched off. Confirmed by assertion and by reading the recorded run, not by an attack trial, on 2026-08-23, over the 90-period scored run and its 90-period control — no attack run was fired; see redteam.why.

Each boundary above was checked by running an assertion, not by an argument. Two of the five were written after a first version convicted the wrong thing — the cumulative check banned the WORD 'cumulative' and convicted all 90 documents on the clause that says the guarantee is NON-cumulative, and the answer-key check banned the substring 'gold' and convicted the floor on its own docstring. Both are recorded in the file.

The result0 attack trials, five boundaries checked by assertion — and the two that matter most on this vertical are properties of what is ABSENT: nothing here issues an invoice and nothing here can reach a licensee.

The Licensee Remarks section IS the field an outside party would influence in a real deployment — a counterparty writes it and the licensor reproduces it verbatim on the statement. It is deliberately sent to the model rather than stripped, because hiding the injection surface would hide it from the run that is supposed to measure it. On THIS corpus it is one of three fixed sentences, so there is nothing adversarial in it and no attack run was fired.

HonestyWhat this does not prove

• Whether a real deployment's Licensee Remarks — prose a counterparty writes freely — would carry an instruction the model follows. Not applicable on this corpus and unmeasured for any other.
• Whether the model's reply can be made to emit something that looks like an invoice instruction. Nothing downstream would act on it, because there is nothing downstream — but the reply itself was never attacked.
• The key-handling claim is checked by reading the code and the UI's error path, not by a leak test over a full run. Nine live calls is not a sample.
• Whether a clause reproduced on a statement could itself be adversarial — a counterparty drafting 'disregard prior instructions' into clause 4.3. Ordinary in principle, absent here, untested.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

No invoice and no contact with the counterparty, non-configurable. This kit produces a status, an invoiceability opinion, an overage figure and a clause citation, and stops.

Everywhere and nowhere — it is a property of what is ABSENT. The only writers in the kit are evals/run.py (results/*.json) and src/state.py (data/state.json). There is no mail path, no webhook, no billing client and no configuration flag that adds one.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• IT IS NOT A CHECK ON WHETHER THE ANSWER IS RIGHT. The balance is correct whatever the reader says, which means a wrong status is reported wrongly for that period and the next period starts from the right number anyway. That is a design decision and it means the kit cannot compound its own errors — it also means it cannot detect them.
• IT IS NOT A LEGAL OPINION. Every clause here is illustrative drafting written for the kit. invoiceable: YES is an opinion about a sentence, and sentences in licence agreements are argued about for a living.
• IT IS NOT A GUARANTEE THAT A CROSSING WAS SEEN. A missed run is not a late alert on this shape — it is NO alert. Skip a quarter and the deal that crossed in it appears next time as ALREADY_IN_OVERAGE, which is the quiet answer.
• IT IS NOT A SCHEDULER. There is no timer, no cron entry and no daemon in this repository. What wakes it, and what happens when a statement is late, is the adopter's decision.
• IT IS NOT EVIDENCE THAT A MODEL HELPS, AND THAT IS NOW MEASURED RATHER THAN SUSPECTED. The scored run tied the free clause-rule floor on every decision grader and lost to it on the governing clause and the overage figure. What the model arm DID establish, which no rule engine could have, is that the carried position is load-bearing for a model too — see the ripple edge against s001.

WatchedWhat is watched, and why that one

The kit measures 62 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

14 measured by the latest run 48 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (ablated False, cells_scored 90, deals 30, documents 90, gold_crossings 25, stateless False) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

7 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

triage · with the model in the path — 6 runs. Columns here are only ever compared with each other.

not a time series No two of these 6 runs measured the same system — they differ on cells_scored, deals, documents, gold_crossings, max_tokens, provider, stateless, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 14 chips that all say so.

DeviationsWhat deviated

0 breaches across 7 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

⚑ THE KIT'S OWN CADENCE, WHICH IS NOT THE SAME QUESTION AS THE MONITOR'S. Re-run evals/check_labels.py (free, nine assertions, no key) on any change to tools/build_corpus.py, src/recoup.py, src/state.py, src/prompt.py or evals/baseline.py — all five can silently invalidate the comparison. Re-run both free floors (free) whenever the corpus is regenerated; they are the published result. Re-run the calibration before any scored run on a new provider or model, because the output ceiling is a per-model fact and this kit has already lost a period to one. ⚠︎ THE MONITORED CADENCE — how often the watch itself should run — is quarterly and is fixed by the agreements rather than by this kit; see environment.ladder's cadence row.

In one lineNo framework, deliberately

No framework and no dependencies at all — json, os, re, random, argparse, urllib, http.server and concurrent.futures, every one of them standard library. requirements.txt is empty on purpose and evals/check_labels.py would convict a floor that imported os.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph, and the one place it looks like one is a straight line. Each deal is a chain of three periods processed in order; the chains are independent of each other and run concurrently. There is no branch, no loop and no tool call — the model fills exactly one slot in a fixed pipeline and code decides everything else.

The other sideWhat a framework costs you

• No scheduler. This is a monitor that carries state and is still INVOKED, not woken. Adopting it means choosing a cadence, and the cadence is the half of this shape the estate keeps skipping.
• No state store. src/state.py is a JSON file replaced atomically. That is right for 30 deals and wrong for 30,000 — and the failure mode of a truncated state file is that it reads as 'no history', which silently resets every balance to zero.
• No retry beyond the adapter's four backoffs. A deal whose call fails is recorded as a failure and its position still advances; nothing re-queues it.
• No observability beyond the result file. There is no metric emitted, no trace and no alert — the kit writes a JSON file and stops.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-mg-recoup on the fast tier, with the carried position, 2026-08-23. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 5 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b000-mg-recoup-ledger, b001-mg-recoup-clauserules recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the population is re-read whole on each scheduled run, and the prior run's carried state is what makes a reading a CHANGE.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Nothing is fetched and no third-party grant is involved; every deal, clause, licensee, title, statement, cash flow and gold label is generated here. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe three-way status, per deal-period, exact match against the computed answer key

For each of the 90 deal-periods, did the reply equal the computed status — UNRECOUPED, CROSSED_THIS_CYCLE or ALREADY_IN_OVERAGE?

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model. The same function grades both free floors and the stub, so no arm is scored more kindly.

Every grader on these pages scored the same 450 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 90. A reply that did not parse counts as a MISS, never as an abstention.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, computed by src/recoup.replay over the planted cash flows at generation time. This grader IS the reference, so its own error rate is not measured — scoring it against itself would be circular.

How tight can the band be? There is no tuned threshold here: the comparison is exact string equality against a closed list of three. The only judgement in the grader is that an unparsed reply is a MISS rather than an abstention, which is stated in the formula rather than swept for.

Cadence: every run, free — and re-derived from a cold start by evals/check_labels.py before any run may spend

The decisionWhen to reach for it

In one lineMissed crossings and false raises, counted apart against different denominators

Of the 25 deal-periods whose guarantee was recouped IN that period, how many were reported as anything else? And of the 65 that did not cross, how many were reported as a crossing?

MethodHow the test was run

evals/scoring.py::score.

Every grader on these pages scored the same 450 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

missed_crossing_pct = missed / 25. false_raise_rate_pct = false / 65. They are NEVER averaged and never combined into an F-score.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl — the same key the status grader is measured against; this grader slices it rather than replacing it. The denominators are 25 and 65 and both are small: one cell is 4.0 points on the first and 1.5 on the second.

How tight can the band be? No sweep and no tuned cut-off: a crossing either is or is not reported as CROSSED_THIS_CYCLE. The denominators are 25 and 65, so one cell is 4.0 points on the first and 1.5 on the second, and no alarm band may be finer than that.

Cadence: every run, free, alongside the status grader

The decisionWhen to reach for it

In one lineStatus accuracy restricted to the 60 cells whose answer is on no page

On the P2 and P3 cells — where the correct answer depends on a balance no statement prints — how many were right?

MethodHow the test was run

evals/scoring.py::score, sliced on gold.memory_dependent.

Every grader on these pages scored the same 450 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 60.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: gold.memory_dependent, asserted by evals/check_labels.py to be exactly (seq > 1) for all 90 rows.

How tight can the band be? The slice is structural rather than tuned: gold.memory_dependent is exactly (seq > 1) and check_labels asserts it on all 90 rows. One cell is 1.7 points on this denominator.

Cadence: every run, free, as a slice of the status grader

The decisionWhen to reach for it

In one lineThe cumulative overage in dollars, exact within one dollar, plus MAE

Did the reply's overage figure match the computed one within one dollar, and how far out was it on average?

MethodHow the test was run

evals/scoring.py::score.

Every grader on these pages scored the same 450 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

exact = |reported - gold| <= 1.00, over 90 cells. MAE is the mean absolute error in dollars over the same 90, with an unanswered cell charged its full gold value.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: gold.overage_cents, integer cents throughout — a running total compared against a threshold cannot be held in float dollars without the crossing period flipping on a representation error.

How tight can the band be? ONE THRESHOLD EXISTS AND IT WAS NOT SWEPT: the one-dollar tolerance. It was chosen because the answer is asked for in whole dollars and the key is held in cents, not because anything measured it — and the kit says so rather than implying a sweep.

Cadence: every run, free, alongside the status grader

The decisionWhen to reach for it

In one lineWhich clause feature governs this deal's recoupment, exact match on five codes

MG_PER_TERM, CROSS_COLLATERALIZED, MG_PER_PERIOD, RESERVE_HELD or COSTS_LIMITED — did the reply name the feature the clause actually carries?

MethodHow the test was run

evals/scoring.py::score.

Every grader on these pages scored the same 450 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 90, 18 cells per code.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: gold.basis, one code per deal, fixed by the clause shape the generator drew.

How tight can the band be? No threshold: exact match on a closed list of five codes, 18 cells each. The base rate being a round 20.0 is what makes this grader readable at a glance.

Cadence: every run, free, alongside the status grader

The decisionWhen to reach for it