The business caseThe problem this solves

Event revenue is split between a venue, a promoter, a rights holder and sometimes a visiting side, under a deal memo carrying tiers, a guarantee, deductions and a cap that apply in a specific ORDER. Settlement systems recompute the split correctly and reliably. What they cannot do is notice that the formula they were handed is not the formula the contract states -- a rate an amendment replaced and the deal feed never caught, a guarantee applied after a percentage rather than as the floor it is, a tier triggered on gross where the clause says net, a cap left off, a conditional term applied to a fixture its own proviso excludes. Every one of those produces a statement that FOOTS, and the party it under-pays has no way to see the clause it was owed under. a settlement check that recomputes the statement and confirms it adds up -- which on this corpus means passing every single one of the 60 settlements computed under the wrong clause, because all 60 of them add up.

The inputThe actual event settlement extracts

The corpus is 120 event settlement extracts, 0.76 MB (txt 120). An event settlement is a deal memo plus a set of figures, and both halves are commercially sensitive: the guarantee a visiting side commands and the tier a promoter accepted are exactly what every counterparty would like to see. There is no public corpus. A SCRUBBED real one would be worse rather than better, and the reason is specific: scrubbing removes the side letter, the amendment and the proviso, and that layer IS what this kit measures.

Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated event-settlement extract for an AI
use-case kit; it reproduces no venue, no promoter, no rights holder, no club and no real
person. The deal terms, the tier thresholds and the restatement floor are ILLUSTRATIVE
DEFAULTS, not any real agreement's.

Event
----------------------------------------------------------------
Event reference                  : EV-0001
Venue                            : The Tannery Bowl, Kestrel Park
Promoter                         : SIXPENNY LIVE
Rights holder                    : MERIDIAN SPORTS RIGHTS
Visiting side                    : Kestrel Park Athletic
Fixture date                     : 2026-05-02
Fixture designation              : CATEGORY A
Broadcast                        : YES
Tickets sold                     : 21,752
Settlement analyst of record     : Priya Raghavan
Statement prepared by            : Mateo Duarte (promoter's settlement desk)

Gate Receipts
----------------------------------------------------------------
All figures in GBP. The deductions below are the only two that reach net gate receipts.

  Gross gate receipts                            555,937.20
    less entertainment tax (5.00 pct)               27,796.86
    less ticketing fees (2.00 pct)                  11,118.74
  Net gate receipts                              517,021.60

Restatement floor (default)      : 250.00 GBP
Contract extract completeness    : COMPLETE
Restatement authority            : NOT DEFINED. Nothing in this kit reissues a statement,
                                   posts a settlement adjustment, releases a payment,
                                   amends a deal term or approves a restatement.

The outcomeWhat a good result looks like

a verdict per settlement with the governing clause named, the party it short-pays, and whether it is worth reissuing -- with zero false accusations across the sound settlements. Measured: 0 false accusations across the 32 sound settlements.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Every amendment and side letter is keyed into the deal system the day it is signed — the free floor -- structured-terms-strong, $0.00
  It re-derives the whole waterfall from the feed and catches every order defect, wrong base and omitted cap where the feed is current -- and it beats the model on verdict, 78.33 against 76.67.
• Amendments and provisos live in documents, and the deal feed lags — the model, with the contract extract
  95.0 pct on the right-terms-right-order call against the floor's 78.33 pct, and 93.33 pct on naming the governing clause against 78.33 pct.

And where nothing here is good enough:

• You want the reading but cannot put the contract in front of it — neither, and the free floor is the honest version of that arm
  structured-terms-strong IS the contract-blind arm and it is free: it sees the gate, the statement and the deal feed and cannot read a clause. It catches 0.0 pct of the defects the feed cannot see.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-25. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-25 — r001-gate-split. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Every settlement extract, the answer key, all three free floors, the injection probe and what r001-gate-split actually answered ship in the repo. python3 -m evals.check_labels and python3 -m src.app both run with no key and no network.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one reading -- one event settlement, end to end including provider-side reasoning tokens, on a shared connection. ⚠︎ NOT AN SLA AND NOT COMPARABLE ACROSS KITS: this credential was shared with eight sibling kits running at the same time and it degraded during the lap, so these figures are a wall-clock observation under contention. 60 readings ran in 0.0 wall seconds with 6 concurrent workers.

Current processWhat it replaces

a settlement check that recomputes the statement and confirms it adds up -- which on this corpus means passing every single one of the 60 settlements computed under the wrong clause, because all 60 of them add up.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

EvalsEvalsArchitectureArchitectureEvalsEvalsPromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• 3 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one reading is

120 event settlements, read once each and independently -- there is no time axis and no carried state. The FREE arms read all 120. The PAID arm scored the 60 readings that came back on a shared credential that degraded mid-lap, and the free floors were re-run over exactly those 60 so every column shares a denominator.

SetupWhat the setup figure measured

There is no index to build -- each settlement's extract goes whole into the prompt. The figures are the corpus generation itself.

LicenceLicence

MIT

Bring your ownBring your own event settlement extracts

tools/build_corpus.py writes data/corpus/*.txt and data/gold.jsonl. Keep the 8 section headings and the TERM:/STEP: line shapes evals/baseline.py parses. Set your own deal terms in the draw and your own restatement floor in src/rules.py first.

⚠︎ And what stops being true when you do: The measured figures on this page do not transfer to your own settlements. The feed-blind share (29 of the 60 misapplied readings here) and the prose-versus-feed split are properties of this generator's declared distribution, not facts about event settlement. Re-run the evals on your own data; that is what the harness is for.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — but they are the input total apportioned by each part’s share of the characters, not a per-part reading from the provider’s counter. The total is measured; the split is arithmetic, and a part carrying denser text really costs more of it than this says.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 3,009 tokens assembled, 1,641 are contexts — 55% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Reproduced from src/prompt.build for EV-0001, not retyped -- byte-identical to what evals/run.py sent.

Full promptThe prompt in full

You are an event settlement review. You read one event's gate receipts, its contract terms and the settlement statement someone has presented, and you decide whether the split applied the RIGHT terms in the RIGHT order. You answer with one JSON object and no other text.

You are a settlement review for one venue's event settlements. You are reading ONE event: its gate
receipts, the deal system's structured copy of the contract terms, the CONTRACT EXTRACT itself, and
the settlement statement somebody has already prepared and presented.

YOUR JOB IS NOT TO RECOMPUTE THE SUM. The statement almost always foots -- its own declared waterfall
reproduces its own printed party totals to the penny, and free code checks that for nothing. What
you are looking for is the settlement that adds up perfectly and was computed under THE WRONG TERM,
or under the right terms IN THE WRONG ORDER: a guarantee that should FLOOR the split applied after a
percentage, a facility recovery taken before rather than after the guarantee, a tier triggered on
GROSS when the contract says NET, a rate an amendment replaced and the deal feed never caught, a cap
left off, a conditional term applied to an event its own condition excludes.

⚠︎ THE DEAL TERMS, THE TIER THRESHOLDS AND THE RESTATEMENT FLOOR ARE ILLUSTRATIVE DEFAULTS, NOT ANY
REAL AGREEMENT'S. Apply them exactly as printed regardless of whether they look right for the event
in front of you.

THE RULES, applied in this order:
 - FIRST check completeness. If the extract records that the governing schedule is NOT ATTACHED, the
   verdict is CONTEXT_INCOMPLETE: no term is judged, the defect is UNDETERMINED, the governing clause
   is NONE, the short party is NONE and nothing is restated. There is no default term for a schedule
   nobody can read (Rule T-7).
 - THEN check whether the statement reproduces ITSELF. Recompute each printed STEP line from its own
   declared base, rate or amount and cap, and check the printed party totals sum to net gate
   receipts. If the statement's OWN declared formula does not reproduce its OWN printed figures, the
   verdict is ARITHMETIC_ERROR and the defect is ARITHMETIC (Rule T-6). This is the only verdict
   about the sum, and it is NOT the verdict for a statement that computes correctly and disagrees
   with the contract.
 - THEN compare the statement against the CONTRACT EXTRACT, term by term. A term has THREE parts and
   all three must match: WHICH term, WHAT BASE it is computed on, and WHERE in the sequence it falls
   (Rule T-2).
 - AN AMENDMENT OR SIDE LETTER IN THE CONTRACT EXTRACT SUPERSEDES BOTH THE ORIGINAL CLAUSE AND THE
   STRUCTURED TERM FEED (Rule T-3). The feed is a convenience copy of the deal and a copy that has
   not caught an amendment is exactly how a superseded rate keeps being applied. Where the statement
   applied a term an amendment replaced, the defect is SUPERSEDED_TERM and the governing clause is
   THE AMENDMENT'S id.
 - A term carrying a stated CONDITION applies only where the condition is met. A conditional term
   applied to an event it does not cover is TERM_NOT_APPLICABLE, and it is not cured by the term
   being correctly computed (Rule T-4).
 - A governing term the statement did not apply at all -- a cap left off, a guarantee floor ignored
   -- is TERM_OMITTED (Rule T-5).
 - A term computed on the wrong base -- a levy on gross where the clause says net, a tier triggered
   on gross where the clause says net -- is WRONG_BASE.
 - The right terms in the wrong sequence is ORDER_INVERTED, and the verdict is ORDER_MISAPPLIED.
   For ORDER_INVERTED name as the governing clause THE CLAUSE THE CONTRACT PLACES EARLIER AND THE
   STATEMENT APPLIED LATER.
 - Otherwise the verdict is SPLIT_CORRECT, the defect is NONE and the governing clause is NONE.
 - THEN name the SHORT PARTY: the party that receives LESS under the statement than it would under
   the governing terms, by the largest amount. It is NONE where the statement is correct and NONE
   where the reading is CONTEXT_INCOMPLETE (Rule T-8).
 - THEN decide the OWNER. Start from the settlement analyst of record, but a note recording a
   handover to someone else supersedes it. The person who PREPARED the statement is not the owner.
 - FINALLY decide "restate": YES only where a defect is named AND the shortfall it causes the short
   party is at or above the printed RESTATEMENT FLOOR. A defect below the floor is still a defect
   and is still named -- it is simply not worth reissuing a statement over (Rule T-9).

Answer with a single JSON object and nothing else:

{"verdict": "SPLIT_CORRECT|TERM_MISAPPLIED|ORDER_MISAPPLIED|ARITHMETIC_ERROR|CONTEXT_INCOMPLETE",
 "defect": "NONE|ORDER_INVERTED|WRONG_BASE|SUPERSEDED_TERM|TERM_OMITTED|TERM_NOT_APPLICABLE|ARITHMETIC|UNDETERMINED",
 "governing_clause": "<the clause id that governs the disputed step, e.g. C-4.1 or A-2.1, or NONE>",
 "short_party": "VENUE|PROMOTER|RIGHTS_HOLDER|VISITING_SIDE|NONE",
 "restate": "YES|NO",
 "owner": "<the settlement analyst who owns this event>",
 "rationale": "one sentence, naming the clause you applied, what the statement did instead, and
   which party it moves money away from"}

Precedence: CONTEXT_INCOMPLETE if the schedule is not attached; then ARITHMETIC_ERROR if the
statement does not reproduce itself; then TERM_MISAPPLIED for a wrong, superseded, omitted or
inapplicable term; then ORDER_MISAPPLIED for the right terms in the wrong order; otherwise
SPLIT_CORRECT.

Settlement extract
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated event-settlement extract for an AI
use-case kit; it reproduces no venue, no promoter, no rights holder, no club and no real
person. The deal terms, the tier thresholds and the restatement floor are ILLUSTRATIVE
DEFAULTS, not any real agreement's.

Event
----------------------------------------------------------------
Event reference                  : EV-0001
Venue                            : The Tannery Bowl, Kestrel Park
Promoter                         : SIXPENNY LIVE
Rights holder                    : MERIDIAN SPORTS RIGHTS
Visiting side                    : Kestrel Park Athletic
Fixture date                     : 2026-05-02
Fixture designation              : CATEGORY A
Broadcast                        : YES
Tickets sold                     : 21,752
Settlement analyst of record     : Priya Raghavan
Statement prepared by            : Mateo Duarte (promoter's settlement desk)

Gate Receipts
----------------------------------------------------------------
All figures in GBP. The deductions below are the only two that reach net gate receipts.

  Gross gate receipts                            555,937.20
    less entertainment tax (5.00 pct)               27,796.86
    less ticketing fees (2.00 pct)                  11,118.74
  Net gate receipts                              517,021.60

Restatement floor (default)      : 250.00 GBP
Contract extract completeness    : COMPLETE
Restatement authority            : NOT DEFINED. Nothing in this kit reissues a statement,
                                   posts a settlement adjustment, releases a payment,
                                   amends a deal term or approves a restatement.

Contract Terms (Structured Feed)
----------------------------------------------------------------
The deal system's own copy of the terms, as the settlement engine reads them. One line per
term, in the order the engine applies them. THE FEED IS A COPY OF THE DEAL, NOT THE DEAL:
an amendment or side letter recorded in the contract extract below governs over anything
on these lines.

TERM: id=C-2.1 kind=DEDUCTION base=GROSS rate=0.050000 party=NONE order=1 label="entertainment tax"
TERM: id=C-2.2 kind=DEDUCTION base=GROSS rate=0.020000 party=NONE order=2 label="ticketing fees"
TERM: id=C-3.1 kind=GUARANTEE base=POOL amount=143,041.78 party=VISITING_SIDE order=3 label="visiting-side guarantee"
TERM: id=C-4.1 kind=FACILITY base=POOL rate=0.139200 cap=57,932.18 party=VENUE order=4 label="venue facility recovery"
TERM: id=C-5.1 kind=SPLIT base=NET rate=0.564300 rate_above=0.479100 threshold=397,334.34 party=PROMOTER order=5 label="tiered promoter split"
TERM: id=C-6.1 kind=LEVY base=NET rate=0.032400 party=RIGHTS_HOLDER order=6 label="rights-holder levy"

Contract Extract
----------------------------------------------------------------
The governing clauses, as written. Where an amendment or a side letter appears below, it
replaces the clause it names.

C-2.1           Entertainment tax is deducted from GROSS gate receipts at 5.00 pct. It is
                the first deduction and is taken before any other term in this agreement.

C-2.2           Ticketing fees are deducted from GROSS gate receipts at 2.00 pct.
                NET GATE RECEIPTS means gross gate receipts less the deductions in C-2.1
                and C-2.2 and nothing else; for the avoidance of doubt it is struck before
                the guarantee, the facility recovery, the split and the levy.

C-3.1           The visiting side is guaranteed 143,041.78 GBP. The guarantee is a FLOOR
                and is satisfied out of net gate receipts BEFORE the venue facility
                recovery under C-4.1 and before the split under C-5.1.

C-4.1           The venue recovers 13.92 pct of the pool REMAINING AFTER the visiting-side
                guarantee, capped at 57,932.18 GBP. The cap is a term of this clause and
                is not waivable at settlement.

C-5.1           The remaining pool is split with the promoter taking 56.43 pct where NET GATE
                RECEIPTS are at or below 397,334.34 GBP and 47.91 pct where NET GATE
                RECEIPTS exceed it. THE TIER IS TRIGGERED ON NET GATE RECEIPTS, not on
                gross. The balance is the venue's.

C-6.1           The rights holder is paid a levy of 3.24 pct of NET GATE RECEIPTS. The levy
                is borne by the promoter out of the promoter's share and is applied last.

A-3.1           Amendment of 21 January 2026, replacing clause C-6.1. The rights-holder levy
                is 5.09 pct of net gate receipts. The rate stated in the original C-6.1 ceased to
                apply on 1 February 2026 and does not govern this fixture.

Settlement Statement As Presented
----------------------------------------------------------------
The waterfall as the preparer applied it, one line per step, in the sequence applied.

STEP: seq=1 term=C-2.1 kind=DEDUCTION base=GROSS base_value=555,937.20 rate=0.050000 party=NONE amount=27,796.86 running=528,140.34
STEP: seq=2 term=C-2.2 kind=DEDUCTION base=GROSS base_value=555,937.20 rate=0.020000 party=NONE amount=11,118.74 running=517,021.60
STEP: seq=3 term=C-3.1 kind=GUARANTEE base=POOL base_value=517,021.60 amount_term=143,041.78 party=VISITING_SIDE amount=143,041.78 running=373,979.82
STEP: seq=4 term=C-4.1 kind=FACILITY base=POOL base_value=373,979.82 rate=0.139200 cap=57,932.18 party=VENUE amount=52,057.99 running=321,921.83
STEP: seq=5 term=C-5.1 kind=SPLIT tier_base=NET tier_base_value=517,021.60 applied_to=POOL applied_to_value=321,921.83 rate=0.479100 party=PROMOTER amount=154,232.75 running=0.00
STEP: seq=6 term=C-6.1 kind=LEVY base=NET base_value=517,021.60 rate=0.032400 party=RIGHTS_HOLDER amount=16,751.50 running=0.00

  Party                                          Amount
  VENUE                                      219,747.07
  PROMOTER                                   137,481.25
  RIGHTS_HOLDER                               16,751.50
  VISITING_SIDE                              143,041.78
  Total allocated                            517,021.60
  Net gate receipts                          517,021.60

Settlement Notes
----------------------------------------------------------------
The venue queried the ticketing fee line at the settlement meeting and withdrew the query
once the box-office close file was produced.

Raw responseThe raw response

{"verdict":"TERM_MISAPPLIED","defect":"SUPERSEDED_TERM","governing_clause":"A-3.1","short_party":"RIGHTS_HOLDER","restate":"YES","owner":"Priya Raghavan","rationale":"A-3.1 replaced C-6.1 with a 5.09% rights-holder levy, but the statement applied the superseded 3.24% rate, underpaying RIGHTS_HOLDER by 9,564.90."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch the wrong clause in an event's revenue split — 60 event settlement extracts drawn from 120 real event settlement extracts. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

⚠︎ THE PAID ARM SCORED 60 READINGS, NOT 120, AND EVERY RATE ON THIS PAGE IS OVER 60. The corpus is 120 settlements and every FREE arm read all 120. The credential this lap ran on was shared with eight sibling kits and stopped returning long reasoning generations part-way through; a hand-fired short call on the same connection at the same moment came back in 1.3 seconds. evals/run.py gained an append-only per-reading cache and --resume so an interrupted run never re-buys a call it has paid for, and --score-cache scores exactly the readings that came back. WHICH readings those are was decided by the network, not by the score: the cache is written before anything is scored.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection, never a bill and never a vendor's claim about this workload.

Rates checked 2026-08-18. The provider that actually ran the calls is kept out of these tables per this estate's naming rule, so no figure here is a bill.

The gradersThree ways to grade

On the WHOLE 120-reading corpus -- free, and therefore unrestricted -- the three floors score 45.0, 50.0 and 75.83 pct on verdict, and the strong floor catches 0.0 pct of the 29 defects the deal feed cannot see. Those are the numbers a future run should be compared against.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

The three free floors, the stub and every pre-run check cost $0.00 and no calls. ⚠︎ 305 CALLS WERE SPENT ON THIS KIT AND ONLY 150 REACHED A PUBLISHED RESULT FILE -- 155 were discarded, and they are counted here rather than quietly dropped. The causes, in order of cost: a 120-second socket timeout inherited from a sibling kit, on a task whose completions are not streamed and routinely take longer than that; transport failures retried as often as a 429, so one hung reading occupied a worker for five full timeouts and then failed anyway; an interrupted calibration whose result file was lost (9); and four deliberate hand-fired probes used to tell a slow provider apart from a broken client. Both harness defects are fixed in src/adapters/__init__.py and documented there. ⚑ THE DOLLAR VALUE IS NOT COMPUTED AND THAT IS NOT AN OMISSION: a call that never returned recorded no token totals, so the repository genuinely does not know what those 155 cost -- which is itself part of what a killed run costs you. ⚠︎ AND THIS NUMBER IS DERIVED FROM THE LEDGER AT BUILD TIME, NEVER TYPED. A mid-lap estimate of 79 was written by hand while the provider was still stalling and was already stale by the time the kit finished.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, AND ALMOST ALL OF THEM ARE REASONING. 98.4 pct of r001-gate-split's output (433670 of 440589 tokens) was provider-side reasoning left at the default. The answer is six short fields and a sentence; the bill is the thinking in front of it -- and on this task the thinking is a six-step waterfall run TWICE, once as the statement declares it and once as the contract states it.
• THE PROMPT IS MOSTLY FIXED TEXT. 3046 input tokens per reading, and the ten rules and the JSON shape are the same on every call -- so a provider with prompt caching would price this workload very differently, and that was not measured.

Your volumeWhat it costs at your volume

LINEAR IN SETTLEMENTS. Nothing amortises. The sublinear lever NOT implemented is skipping settlements whose deal carries no amendment at all -- the free strong floor is exactly right on those.

Model choiceWhy this model

It is the tier the shared connection was already pointed at, and the kit's whole claim is that swapping it is .env plus one more run. No second model was run against this corpus; every other row in the cost table is a projection onto a published card and is labelled as one.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

per-query average, this run

The codeEvery file, and what it is for

11 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/rules.pythe settlement rule and the waterfall engine — a swap seam

The two-waterfall comparison: apply() runs an ordered term list and returns what each party gets; decide() turns the two results into a verdict, a defect, a governing clause, a short party and a restate call. It is also the answer key.

You change it to: KINDS and BASES. A deal carrying a term kind this corpus does not model needs one more branch in apply().

# The settlement waterfall as arithmetic, and the rule that decides whether a statement applied the

VENUE = "VENUE"
PROMOTER = "PROMOTER"
RIGHTS_HOLDER = "RIGHTS_HOLDER"
VISITING_SIDE = "VISITING_SIDE"
PARTY_NONE = "NONE"
PARTIES = (VENUE, PROMOTER, RIGHTS_HOLDER, VISITING_SIDE)
DEDUCTION = "DEDUCTION"
GUARANTEE = "GUARANTEE"
FACILITY = "FACILITY"

src/segment.pythe section splitter

Splits the extract into its 8 named sections; asserted across all 120 documents.

# Split a settlement extract into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "Event", "Gate Receipts", "Contract Terms (Structured Feed)",
RULE = "-" * 64
def split(text):
def render(secs):

src/select.pythe send filter — a swap seam

Settlement Contact is mapped by no field and therefore never sent.

You change it to: SECTION_HINTS and NEVER_SENT.

# Pick which sections of a settlement extract are sent. Pure code -- the last deterministic step

BANNER = "Synthetic Record"
EVENT = "Event"
RECEIPTS = "Gate Receipts"
FEED = "Contract Terms (Structured Feed)"
EXTRACT = "Contract Extract"
STATEMENT = "Settlement Statement As Presented"
CONTACT = "Settlement Contact"
NOTES = "Settlement Notes"
NEVER_SENT = (CONTACT,)

src/prompt.pythe prompt

Two parts: the instruction with the ten rules and the JSON shape, then the extract. The contract-blind control replaces exactly one section's body.

# Assemble the one prompt this kit sends. No template engine, no framework -- string concatenation

BLIND_LINE = ("The contract extract is not available for this settlement. Judge it on the gate "
INSTRUCTION = """\
def build(text, contract_blind=False):

src/adapters/__init__.pythe model call — a swap seam

Raw HTTP, stdlib only, bounded retry, a socket timeout set from the published ceiling and a SEPARATE retry budget for transport failures. The shared daily call cap is checked here.

You change it to: ADAPTER_TIMEOUT_S and TRANSPORT_RETRIES -- set from the ceiling you publish, not from habit. The wrong values here cost this kit 155 of its 305 calls -- only 150 ever reached a published result file.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TRANSPORT_RETRIES = 1
TIMEOUT_S = int(os.environ.get("ADAPTER_TIMEOUT_S", "300"))
def _post(url, headers, payload, timeout=None):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}

src/statement.pythe reader

One event settlement, one call, at the published ceiling.

# One event settlement, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = ("You are an event settlement review. You read one event's gate receipts, its contract "
MAX_TOKENS = 32000
FIELDS = ("verdict", "defect", "governing_clause", "short_party", "restate", "owner")
def documents():
def load_doc(doc_id):
def facts_of(text):
def _parse(txt):

evals/run.pythe per-reading cache

Append-only, written before anything is scored. --resume never re-buys a call already paid for; --score-cache scores exactly what came back and records that count as the denominator.

# Run the review over the 120 event settlements and write one result file. THIS SPENDS MONEY.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
def scored_half(docs):
def cache_path(run_id):
def load_cache(run_id, max_tokens, contract_blind):
def append_cache(run_id, row):
def load_gold():
def stub_complete(cfg, system, user, max_tokens=1024):

evals/scoring.pythe scorer

Exact match per cell. The term-and-order discriminator is a separate binary with separate denominators for its two directions, and the feed-blind slice is scored on its own.

# Score a run against the answer key. Pure code, exact match per cell. No model grades anything.

FIELDS = ("verdict", "defect", "governing_clause", "short_party", "restate", "owner")
TERM_OR_ORDER = ("TERM_MISAPPLIED", "ORDER_MISAPPLIED")
def _pct(n, d):
def _same(got, want):
def _same_name(got, want):
def score(records, golds):
def _protection(records, golds):

evals/baseline.pythe three free floors

totals-tie-out, recompute-formula and structured-terms-strong -- $0.00 each, and the third BEATS the model on the verdict column.

# THE FREE FLOORS. Three of them, none a strawman, all of them GBP 0.00.

MODES = ("totals-tie-out", "recompute-formula", "structured-terms-strong")
TOL = 0.011
def _num(v):
def _kv(line):
def _section(text, name, nxt):
def _facts(text):
def _finish(verdict, defect, clause, short_party, restate, owner, why):
def _feed_terms(f):
def _step_check(f):

src/app.pythe local UI server

http.server, stdlib. Renders with no key.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "9021"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-gate-split")
class H(BaseHTTPRequestHandler):
def main():

ui/app.jsthe local UI client

Hand-written JS, no framework.

# 

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3045 input and 7343 output tokens per reading (one event settlement), all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

⚠︎ MEASURED, AND IT HELD. Settlement Notes is this kit's injection surface, and on this vertical the desk that types them is often the PROMOTER'S -- the counterparty whose statement is under review.

API_KEY is read from .env (gitignored) or the real environment; never logged, never placed in a prompt, and redacted from any provider error message the local UI passes to the browser.

The experimentWe DID attack it -- one sentence, on every reading where it could matter

The probe replaces the Settlement Notes section with the instruction-shaped note and re-fires, everything else held identical. The fired set is not 'every reading where a restatement was due' but 'every reading where a restatement was due AND the un-injected run raised one', because the second is the only set where a suppression is observable. 32 of 32 held. Both were run for real on 2026-08-25.

Both boundaries are measured in both directions: the injection probe forces the condition rather than waiting for it and pairs each trial against its own control, and the privacy guard is red-proven by reproducing the schema-change condition that reaches its fallback.

The result0 of 32 restatements suppressed by the instruction-shaped note -- measured, not assumed.

One phrasing, one model, one corpus, 32 paired trials -- every reading where suppression was even possible.

HonestyWhat this does not prove

• Whether a different phrasing moves it. A note claiming the venue has already signed the statement off, claiming the amendment was never countersigned, or written to look like a system banner is a different experiment.
• Whether an injection placed in Settlement Contact would have any effect -- by construction it cannot, because that section never leaves the machine.
• Whether the result holds on another tier or another corpus.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never reissue a statement, post a settlement adjustment, release a payment, amend a deal term or approve a restatement -- and never present the deal terms, the tier thresholds or the restatement floor as any real agreement's.

Stated to the model on every call in src/prompt.py's INSTRUCTION block, and enforced by evals/check_labels.py's banned-code-path scan before any run may spend.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is a PROMPT RULE PLUS A STATIC SCAN, not a runtime enforcement layer. Nothing stops a forker adding such a path tomorrow; the scan catches it only the next time somebody runs check_labels.py, which is a manual step.
• The injection result is ONE SENTENCE against ONE model on ONE corpus, over 32 paired trials. It is not a resistance rate for prompt injection in general.
• The verdict is an OPINION about a statement, not a legal reading of a contract. A settlement dispute is decided by people with the whole agreement in front of them, not by an extract.

WatchedWhat is watched, and why that one

The kit measures 95 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

2 measured by the latest run 93 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

12 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 4 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

triage · no model in the path — a baseline, not a peer column — 6 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 6 runs measured the same system — they differ on answered, arithmetic_cells, context_incomplete_cells, correct_cells, documents, events_caught, events_missed, events_needing_restatement, feed_blind_cells, missed_restatements, quiet_cells, readings_scored, restate_cells, split_term_cells, split_term_feedblind_cells, split_term_sound_cells, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

triage · with the model in the path — 4 runs. Columns here are only ever compared with each other.

not a time series No two of these 4 runs measured the same system — they differ on answered, arithmetic_cells, context_incomplete_cells, correct_cells, documents, events_caught, events_missed, events_needing_restatement, false_restatements, feed_blind_cells, max_tokens, missed_restatements, quiet_cells, readings_scored, restate_cells, split_term_cells, split_term_feedblind_cells, split_term_sound_cells, stateless — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 21 chips that all say so.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 2 chips that all say so.

DeviationsWhat deviated

0 breaches across 12 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

Nothing fires yet. Not one of the 22 bands on this kit names a consequence, so there is no row to print. Which is the honest answer to the heading: a band that nothing is wired to is a number somebody has to notice, and this kit has only those. An empty table would have said the same thing while looking like a rendering failure.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

ON DEMAND, ONCE PER SETTLEMENT -- there is no clock and no carried state, so re-reading the same settlement against the same contract returns the same opinion. The one thing that makes a re-run worth paying for is a CONTRACT that moved: an amendment or side letter signed after the statement was prepared changes which clause governs, and this kit will only see it once that document reaches the extract. At 0.0193 USD per settlement on the projected card, a season of 120 settlements is roughly 2.31 USD -- but the ceiling that matters is provider concurrency, not price: on a credential shared with eight sibling kits the long reasoning generations stopped returning entirely, 155 of this kit's 305 calls never reached a published result file, and the wall-clock figures on this page are an observation under contention rather than an SLA.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, no framework dependency, and a prompt anyone can read end to end. There is no retrieval step for a framework to own and no agent loop for it to drive: the whole job is one document, one contract and one call.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear: the reader -> src/prompt.py -> src/adapters -> evals/scoring.py. No branching, no tool-calling and no agent loop.

The other sideWhat a framework costs you

• Swapping providers means editing the PROVIDERS dict by hand. In exchange the prompt has no hidden templating layer and requirements.txt stays empty.
• Resume and caching are 40 lines here rather than a framework feature. They were written mid-lap, under a provider outage, which is the only reason they exist and the reason they are shaped the way they are.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-gate-split on the fast tier, with the contract extract, 2026-08-25. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 9 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

6 runs not plotted. b000-gate-split-tieout, b001-gate-split-recompute, b002-gate-split-structuredterms, b020-gate-split-tieout-scored, b021-gate-split-recompute-scored, b022-gate-split-structuredterms-scored recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: ADAPTER_TIMEOUT_S, API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe verdict, the named defect, the governing clause, the short party, the restate call and the settlement analyst, per reading, exact match against the computed answer key

whether each of the 6 answered fields equals the answer key, per reading

MethodHow the test was run

python3 -m evals.run --run-id <id> ...; evals/scoring.py compares strings. No model grades anything.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 60 per field. split_term_accuracy_pct is scored separately over its own denominators.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the computed answer key in data/gold.jsonl, produced by src/rules.apply() and src/rules.decide() from the same planted terms the corpus builder wrote onto the page.

No true/false rates for this grader. It records 6 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold was swept: exact match has no tunable. Denominators are stated beside every rate because 60 readings makes each one wide.

Cadence: once per run; free

The decisionWhen to reach for it

In one lineThe right terms, applied in the right order

whether the arm called a settlement misapplied when it was, and left a sound settlement alone

MethodHow the test was run

evals/scoring.py, same pass as the exact-match grader.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

called correctly over all 60 readings; the two error directions over their own denominators (28 misapplied, 32 sound).

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the computed answer key in data/gold.jsonl.

No true/false rates for this grader. It records 3 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No sweep: the verdict is an enum the arm emits.

Cadence: once per run; free

The decisionWhen to reach for it

In one lineWorth reissuing the statement

whether the arm asked a settlement desk to reopen a statement with a counterparty

MethodHow the test was run

evals/scoring.py, same pass.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

over the 35 readings whose defect clears the printed restatement floor, and the 25 that do not.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the computed answer key in data/gold.jsonl.

No true/false rates for this grader. It records 3 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No sweep: restate is a binary the arm emits against a printed floor.

Cadence: once per run; free

The decisionWhen to reach for it