The business caseThe problem this solves

A signed sponsorship agreement does not arrive as a list of what the property owes. It arrives as a rights schedule — a table of assets and quantities — and a stack of clauses, and somebody has to turn it into an inventory before the season starts. The table is the trap. Every row on it looks like a promise, and several of them are not: a right that only arises in a renewal term nobody has exercised, a make-good that only exists if the property misses something else, a category exclusivity the agreement promises to try for rather than grant, and an asset the schedule names with a quantity and no clause anywhere saying the property will provide it. Get any of those wrong in the optimistic direction and the property has published a promise it never made; get the fulfilment window wrong and it has published a promise nobody can plan. Someone opening each signed agreement, typing the rights schedule into a fulfilment tracker, and then reading the clause behind every row for two things the agreement may or may not settle — whether the property owes the asset in this term at all, and when it comes due. The free floor shipped with this kit is that person's spreadsheet, and it is measured: it reads every stated fact correctly and still books 202 of 202 not-owed assets as owed.

The inputThe actual sponsorship agreements

The corpus is 50 sponsorship agreements, 0.15 MB (txt 50). Plain text, one format, invented rather than fetched — a real sponsorship agreement names a real property and a real sponsor, states a rights fee both sides treat as commercially confidential, and its asset schedule is exactly what a rights-holder's competitors would most like to read. There is no public corpus of (agreement, correct deliverable worksheet) pairs. Generating it also makes the label mechanical: gold's two calls are the shipped rulebook's own lookup run over the same four facts the pack itself states, never somebody's reading of what the deal really was. The composition is exact and dealt rather than sampled — every bucket is a fixed count allocated over the slots whose asset category can carry it, and the allocator raises rather than back-filling.

Agreement
---------
SPN-0001

Fee And Payment Reference
-------------------------
Sponsor Dunhollow Beverages, regional partner of Thornmoor Invicta. Sponsor account SA-93372,
signatory reference SG-2528. Rights fee USD 391,000 for the 2026-27 season, payable in 2
instalments.

Rights Schedule
---------------
Agreement AG-73967, executed 2026-04-12, for the 2026-27 season. Assets as scheduled:

  AS-9038   Concourse digital display network              181 units
  AS-3986   Category exclusivity, quick-service dining     category-wide
  AS-5131   Commentary-position backdrop branding          32 spots
  AS-9180   Sampling rights, outer concourse               101 sqm
  AS-6160   Tunnel-mouth backdrop branding                 232 units
  AS-2087   Fan-zone activation space, concourse east      105 sqm
  AS-8642   Newsletter placement, weekly edition           STRUCK -- side letter SL-3

Asset AS-9038
-------------
Description: Concourse digital display network
Category: signage
Scheduled quantity: 181 units
Scheduled against each matchday of the 2026-27 season.
The 2026-27 season runs to 17 matchdays.

Asset AS-3986
-------------
Description: Category exclusivity, quick-service dining
Category: category_exclusivity
Scheduled quantity: category-wide
The Property shall use reasonable endeavours to make the asset described above available.
Scheduled across the 2026-27 season as a whole.

Asset AS-5131
-------------
Description: Commentary-position backdrop branding
Category: broadcast
Scheduled quantity: 32 spots
The Property intends to make the asset described above available.
Attaches to each matchday of the 2026-27 season.
The matchdays are those in the schedule issued by the organiser and not appended to this
agreement.

Asset AS-9180
-------------

The outcomeWhat a good result looks like

A worksheet row per scheduled asset — the asset code, its description, its category, the four facts the agreement states about it, and the two calls the shipped rulebook makes — plus one pure-code review flag per agreement. The column that earns the page is the one that says not_determined: 100 of the 322 rows in this corpus are things the agreement never actually promises, and 125 of them attach to no fulfilment window the agreement names.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Turning a signed season's worth of sponsorship agreements into a deliverable inventory before the first fixture — the one tier measured here — 322 of 322 standing calls, 317 of 322 windows, and 0 of 202 not-owed assets booked as owed
  100 pct on the standing call against the free tracker floor's 37.27 pct, and 0 booked against the floor's 202. It also left every one of the 42 decoy codes off, against the floor's 29 phantoms. Median 21.9 seconds an agreement, $0.0145098 a pack on the shared published card.
• Deciding whether to trust the not_determined column at all — the determinacy split and the booked-as-owed count, read together
  2.22 pct over-confidence with 0 pct over-caution says the caution is not being bought with noise, and 0 of 202 booked says it is not being bought by declining everything either. The free floor is the control: 100 pct over-confident and 202 of 202 booked.

And where nothing here is good enough:

• Deciding whether a cheaper or a more deliberate model would do better here — nothing — this page cannot answer it
  ONE MODEL WAS RUN ONCE. There is no second tier and no repeat run, so there is no separation to report and no variance estimate. The one comparison on this page is model against free floor, which is a comparison of approaches rather than of models.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-sponsor-deliverables. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts on 127.0.0.1:8961, the 50-agreement picker populates, the worksheet draws its six column headings and the decided panel its five rows, the shipped rulebook renders from data/rulebook.json, and Build worksheet returns a sentence saying nothing was called rather than an error. requirements.txt is empty — standard library only — so there is no install step before any of that works. python -m evals.check_labels and both free runs (--stub, --baseline) complete with no key.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per agreement pack

Current processWhat it replaces

Someone opening each signed agreement, typing the rights schedule into a fulfilment tracker, and then reading the clause behind every row for two things the agreement may or may not settle — whether the property owes the asset in this term at all, and when it comes due. The free floor shipped with this kit is that person's spreadsheet, and it is measured: it reads every stated fact correctly and still books 202 of 202 not-owed assets as owed.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings; a pack with none falls back to one whole-document segment

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 50 packs cut into 564 sections in about nine thousandths of a second, in process, with no model and no network. 514 of those sections are sent and the other 50 are the Fee And Payment Reference sections, which no field maps to.

LicenceLicence

MIT — this repository's own licence. Every sponsor name, property name, agreement reference, asset code, commercial category, quantity and fee is invented; no real property, team, league, federation, venue, event, sponsor, brand, product or agency is named anywhere, and no governing body's regulation, industry standard, measurement methodology or rate card is reproduced, paraphrased or named.

Bring your ownBring your own sponsorship agreements

REPLACE data/rulebook.json FIRST. It is this kit's own construction, it reproduces no governing body's rule and no industry standard, and everything downstream reads it: src/rulebook.py loads it, src/prompt.py renders it into every call, tools/build_corpus.py writes gold with it and evals/judge.py re-runs it over the reply. Then replace data/corpus/*.txt with your own packs, write data/fields.json to your own columns, and supply a gold row per pack. Expect to edit SECTION_HINTS in src/select.py for your layout — when it matches nothing, selection falls back to the whole document, which is slower, more expensive and always correct. If your agreements are unlabelled, the cell grade, the identification matrix and both call graders go away and the CONSISTENCY DIAGNOSTIC does not: it re-runs the rulebook over each row's own facts with no gold at all. It caught 243 of 249 of the free floor's call errors and 0 of 5 of the model's, and that asymmetry is the honest statement of what it is worth.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 4,184 tokens assembled, 1,283 are rules — 31% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on SPN-0028 — the same pack the worked example above was captured on, so the four parts sum to exactly the 4184 input tokens that call reported. Four calls at max_tokens=1, each the previous prefix plus one more part; each part's size is the difference between two consecutive prompt_tokens counts the provider itself returned. No local tokenizer is used or trusted.

Full promptThe prompt in full

You read a SPONSORSHIP AGREEMENT PACK -- a rights schedule plus the clause behind each of its assets -- and return a REVIEWER'S WORKSHEET OF DELIVERABLES: what the property owes this sponsor, and what the agreement leaves open. You return JSON and nothing else.

You are preparing a worksheet for a qualified reviewer. You never determine that an obligation was met or missed, never count what was delivered, never value an asset, never calculate a shortfall, never award or size a make-good and never raise an invoice. Your job is to say what the agreement states, apply the rulebook given below, and NAME THE ROWS THE AGREEMENT DOES NOT SETTLE.

RULES, in order of importance:
1. ONE OBJECT PER SCHEDULED ASSET ROW OF THIS AGREEMENT'S RIGHTS SCHEDULE, AND NO OTHERS. A row struck by a side letter is not scheduled, even though its clause is still printed in the pack. An item the SPONSOR supplies to the property is an obligation running the other way and is not a row. An asset another commercial partner holds, listed in the exclusivity annex for context, is not a row. All three carry codes in the same format as a real asset, and putting one on the worksheet puts a promise in front of a rights-holder that the property does not owe.
2. REPORT WHAT THE PACK STATES. `wording`, `precondition`, `cadence` and `occurrence_basis` are readings of the paperwork, not judgements about it. If an asset's clause says nothing about a precondition, `precondition` is 'none_stated' -- that is a finding about the agreement, and it is the most common one. Never infer a precondition or a promise from the asset category, from the quantity, or from what you know about how sponsorships are usually written.
3. `standing` IS DECIDED ONLY BY THE RULEBOOK BELOW, from `wording` and `precondition`. Work through the five steps IN ORDER and STOP at the first that fires:
   a. precondition 'renewal_only' -> 'not_owed_this_term'. This holds WHATEVER the wording says, including an asset written as a full grant.
   b. precondition 'on_shortfall' -> 'remedy_only'.
   c. wording 'endeavour' -> 'not_determined'.
   d. wording 'grant' -> 'owed_this_term'.
   e. anything else -> 'not_determined'.
4. A QUANTITY IS NOT A PROMISE. A schedule row naming an asset and a number, whose clause states no verb of promise at all, is 'not_determined'. It is not 'owed_this_term'. This is the step a confident reader skips, and on this rulebook it is the wrong answer rather than a defensible one.
5. `window` IS DECIDED ONLY from `cadence` and `occurrence_basis`, in order: cadence 'not_stated' -> 'not_determined'; 'term_period' -> 'across_term'; 'single_milestone' -> 'one_off'; 'per_occurrence' WITH occurrence_basis 'count_stated' -> 'each_occurrence'; 'per_occurrence' with anything else -> 'not_determined'. A reference to a fixture list the agreement does not attach is not a count. `window` is asked of every asset independently of its standing call.
6. 'not_determined' IS A REAL ANSWER AND YOU ARE EXPECTED TO USE IT. A worksheet that never reaches for it is guessing, and a right recorded as owed that was never granted is a promise the property will be held to.
7. Copy `asset_code`, `asset_label` and `asset_category` verbatim from the pack. Use the exact allowed value for every field that lists them, and return every field for every asset.

DELIVERABLE WORKSHEET RULEBOOK (the authority for `standing` and `window`; this is an ILLUSTRATIVE rulebook written for this kit, and it reproduces no governing body's regulation, no league or federation rule and no industry standard)

WHAT COUNTS AS A ROW ON THE WORKSHEET
  - An asset is on the worksheet if and only if it is a ROW OF THIS AGREEMENT'S RIGHTS SCHEDULE, running from the property to the sponsor.
  - A row struck from the rights schedule by a side letter is NOT on the worksheet, even though its code and its whole asset clause are still printed in the pack.
  - An item the SPONSOR supplies to the property is NOT on the worksheet. It is an obligation running the other way, and it carries a code in the same format.
  - An asset named in the exclusivity annex because ANOTHER partner holds it is NOT on the worksheet. It is somebody else's inventory, listed here so this sponsor's category can be read against it.

THE FOUR STATED FACTS, read off the agreement
  wording:
    grant                  the asset clause states the property GRANTS, SHALL PROVIDE, SHALL DELIVER or SHALL PROCURE the asset
    endeavour              the asset clause states only that the property will use reasonable or commercially reasonable efforts, will seek to, will endeavour to, or intends to do something
    not_stated             the asset clause describes the asset and states no verb of promise at all - it is an inventory entry, not a sentence about what the property will do
  precondition:
    renewal_only           the agreement states this asset arises only in a renewal, extension or option term that the sponsor has not exercised
    on_shortfall           the agreement states this asset arises only if the property fails to deliver something else - a make-good
    none_stated            the agreement states no precondition for this asset
  cadence:
    per_occurrence         the clause attaches the asset to each occurrence of a repeating event - each home fixture, each matchday, each round
    term_period            the clause attaches the asset to a period, season or term as a whole
    single_milestone       the clause attaches the asset to one named moment - a launch, an announcement, a single appearance
    not_stated             the clause states no fulfilment window of any kind
  occurrence_basis:
    count_stated           the agreement states how many occurrences there are
    schedule_referenced    the agreement points at a fixture list, calendar or schedule that is NOT attached to it
    not_applicable         the asset is not attached to a repeating occurrence, so the question does not arise
    not_stated             the asset IS attached to a repeating occurrence and the agreement fixes no number for it

STANDING -- work through IN ORDER, stop at the first that fires
  1. A RIGHT THAT HAS NOT VESTED IS NOT OWED THIS TERM. If precondition is `renewal_only`, the worksheet records `not_owed_this_term` - whatever the wording says. A renewal-year asset written as a full grant is still a renewal-year asset.
  2. A MAKE-GOOD IS A REMEDY, NOT A DELIVERABLE. If precondition is `on_shortfall`, the worksheet records `remedy_only`. The property owes it if and only if it misses something else, so putting it on the owed list counts the same asset twice.
  3. AN EFFORTS PROMISE NAMES NO DELIVERABLE. If wording is `endeavour`, the worksheet records `not_determined`. `Reasonable efforts to procure` is a statement about conduct, not about a thing the property must hand over, and this worksheet cannot tell you what conduct discharges it.
  4. A GRANT IS OWED. If wording is `grant`, the worksheet records `owed_this_term`.
  5. ANYTHING ELSE IS `not_determined`. A QUANTITY IS NOT A PROMISE. A schedule row that names an asset and a number, with no clause stating the property will provide it, does not settle whether the property owes it. This is the branch a confident reader skips - the row looks like an inventory line, so it gets read as one.

FULFILMENT WINDOW -- work through IN ORDER, stop at the first that fires
  1. If cadence is `not_stated`, the worksheet records `not_determined`. The agreement never says when this is delivered, and a deliverable with no fulfilment window is the single most common thing a sponsorship agreement leaves open.
  2. If cadence is `term_period`, the worksheet records `across_term`.
  3. If cadence is `single_milestone`, the worksheet records `one_off`.
  4. If cadence is `per_occurrence` AND occurrence_basis is `count_stated`, the worksheet records `each_occurrence`.
  5. If cadence is `per_occurrence` and the agreement fixes NO number of occurrences, the worksheet records `not_determined`. `At each home fixture` reads like a settled cadence and is not one: an agreement that points at a fixture list it does not attach has not said how many times the property must deliver. This is the second branch a confident reader skips.
  The window question is asked of every asset independently of its standing call - a renewal-year asset and a make-good still either state a fulfilment window or do not.

WHY `not_determined` IS A REAL ANSWER
  A deliverable worksheet that never says `not_determined` is not a confident worksheet, it is a guessing one. The whole value of this shape is that an account team can see, in one column, which lines the agreement actually settles and which ones somebody has to go back to the sponsor about BEFORE the season starts rather than after a shortfall. Over-confidence here costs more than caution in both directions: a right recorded as owed that was never granted is a promise the property will be held to, and a right recorded as owed with no window is a promise nobody can plan against.

Return these:
- agreement_id (string) -- the agreement identifier from the Agreement section, verbatim
- deliverables (array of objects) -- one object per SCHEDULED asset row, each carrying:
    - asset_code (string) -- the asset code from the Rights Schedule row, verbatim (for example AS-4417). THIS IS THE KEY -- one object per SCHEDULED asset row and no others
    - asset_label (string) -- the asset's description, copied verbatim from its own Asset section's Description line
    - asset_category (enum) one of: signage, broadcast, hospitality, activation, category_exclusivity, digital, ip_licence -- the asset category, copied verbatim from the Category line of the asset's own Asset section
    - wording (enum) one of: grant, endeavour, not_stated -- how the asset's own clause words the promise. 'grant' when it says the property grants, shall provide, shall deliver or shall procure the asset. 'endeavour' when it says only that the property will use reasonable or commercially reasonable efforts, will seek to, will endeavour to, or intends to. 'not_stated' when the clause describes the asset and states no verb of promise at all. Report what the paperwork says -- this is NOT on its own what decides `standing`
    - precondition (enum) one of: renewal_only, on_shortfall, none_stated -- what the asset's own clause STATES about a precondition attaching to it. 'renewal_only' when it says the asset arises only in a renewal, extension or option term. 'on_shortfall' when it says the asset arises only if the property fails to deliver something else -- a make-good. 'none_stated' when the clause states NEITHER. Never infer a precondition from the asset category or from what you know about how sponsorships are usually written
    - cadence (enum) one of: per_occurrence, term_period, single_milestone, not_stated -- what the asset's own clause STATES about when it is delivered. 'per_occurrence' when it attaches the asset to each occurrence of a repeating event -- each home fixture, each matchday, each round. 'term_period' when it attaches it to a period, season or term as a whole. 'single_milestone' when it attaches it to one named moment. 'not_stated' when the clause states no fulfilment window at all. Silence is a finding, not a gap
    - occurrence_basis (enum) one of: count_stated, schedule_referenced, not_applicable, not_stated -- ONLY meaningful when `cadence` is 'per_occurrence'. 'count_stated' when the agreement states how many occurrences there are (a number of fixtures, matchdays or rounds). 'schedule_referenced' when it points at a fixture list, calendar or schedule that is NOT attached to the agreement. 'not_stated' when the asset repeats and the agreement fixes no number anywhere. Use 'not_applicable' for every row whose cadence is NOT 'per_occurrence'
    - standing (enum) one of: owed_this_term, not_owed_this_term, remedy_only, not_determined -- the worksheet's standing call, decided STRICTLY by the shipped rulebook from `wording` and `precondition` and nothing else. Work through the five steps IN ORDER, stopping at the first that fires. (1) precondition 'renewal_only' -> 'not_owed_this_term', WHATEVER the wording says. (2) precondition 'on_shortfall' -> 'remedy_only'. (3) wording 'endeavour' -> 'not_determined'. (4) wording 'grant' -> 'owed_this_term'. (5) anything else -> 'not_determined'. A QUANTITY IS NOT A PROMISE: a schedule row naming an asset and a number, whose clause states no verb of promise, is 'not_determined', never 'owed_this_term'
    - window (enum) one of: each_occurrence, across_term, one_off, not_determined -- the worksheet's fulfilment-window call, decided STRICTLY from `cadence` and `occurrence_basis`. (1) cadence 'not_stated' -> 'not_determined'. (2) 'term_period' -> 'across_term'. (3) 'single_milestone' -> 'one_off'. (4) 'per_occurrence' AND occurrence_basis 'count_stated' -> 'each_occurrence'. (5) 'per_occurrence' with anything else -> 'not_determined', because a reference to a fixture list the agreement does not attach is not a count. Asked of every row independently of its standing call

Return a JSON object with exactly these top-level keys: agreement_id, deliverables
`deliverables` is an array. Return it empty only if the rights schedule has no scheduled asset row at all.

SPONSORSHIP AGREEMENT PACK
--------------------------
Agreement
---------
SPN-0028

Rights Schedule
---------------
Agreement AG-68531, executed 2026-03-04, for the 2027-28 season. Assets as scheduled:

  AS-1283   Concourse digital display network              225 units
  AS-5648   Category exclusivity, domestic appliances      category-wide
  AS-3773   Pre-match interview backdrop placement         43 spots
  AS-2371   Primary mark licence for sponsor advertising   term-wide licence
  AS-3819   App home-screen placement panel                21 placements
  AS-8297   Away-fixture travel allocation, main party     14 seats
  AS-4153   Owned-channel post series, match previews      13 placements
  AS-3556   Tunnel-mouth backdrop branding                 201 units
  AS-5697   Hospitality seats, executive level             STRUCK -- side letter SL-2

Asset AS-1283
-------------
Description: Concourse digital display network
Category: signage
Scheduled quantity: 225 units
The Property shall use commercially reasonable efforts to make the asset described above
available.
This asset is available only as a make-good against a shortfall on another scheduled asset.
Scheduled across the 2027-28 season as a whole.

Asset AS-5648
-------------
Description: Category exclusivity, domestic appliances
Category: category_exclusivity
Scheduled quantity: category-wide
The Property intends to make the asset described above available.
Attaches to the initial term as a whole.

Asset AS-3773
-------------
Description: Pre-match interview backdrop placement
Category: broadcast
Scheduled quantity: 43 spots
Scheduled against each home fixture of the 2027-28 season.
The home fixtures are those in the published fixture list, which is not attached to this
agreement.

Asset AS-2371
-------------
Description: Primary mark licence for sponsor advertising
Category: ip_licence
Scheduled quantity: term-wide licence
The Property shall deliver the asset described above.
Scheduled for the partnership announcement.

Asset AS-3819
-------------
Description: App home-screen placement panel
Category: digital
Scheduled quantity: 21 placements
Attaches to the sponsor's own launch event, once.

Asset AS-8297
-------------
Description: Away-fixture travel allocation, main party
Category: hospitality
Scheduled quantity: 14 seats
This asset applies only if the sponsor exercises the extension option for the following
season, which it has not done.
Scheduled against each home fixture of the 2027-28 season.
See the season calendar maintained by the competition organiser; it is not attached here.

Asset AS-4153
-------------
Description: Owned-channel post series, match previews
Category: digital
Scheduled quantity: 13 placements
The Property shall procure the asset described above.
Attaches to the initial term as a whole.

Asset AS-3556
-------------
Description: Tunnel-mouth backdrop branding
Category: signage
Scheduled quantity: 201 units
The Property grants the sponsor the asset described above.
Scheduled against each matchday of the 2027-28 season.
A total of 23 matchdays are played in the 2027-28 season.

Asset AS-5697
-------------
Description: Hospitality seats, executive level
Category: hospitality
Scheduled quantity: 17 seats
This clause was drafted before side letter SL-2 and is retained in the pack for the audit
trail.

Sponsor-Supplied Items
----------------------
The following is supplied BY THE SPONSOR to the property. It is not an asset the property
provides to the sponsor and it is listed here for completeness only:

  AS-5166   Sampling stock for the fan zone                10 crates per fixture

Exclusivity Annex
-----------------
Asset AS-8012, Category exclusivity, retail banking, is held by another commercial partner of
the property. It is listed here so that this sponsor's category can be read against it, and it
is not provided to this sponsor.

Agreement Notes
---------------
This pack is a draft for the account team's review.
Side letter SL-2 removed asset AS-5697 from the rights schedule before execution; it is not
provided under this agreement.
Circulated for review only. No fulfilment record exists against this schedule and no shortfall
or make-good has been assessed.

Raw responseThe raw response

{"agreement_id":"SPN-0028","deliverables":[{"asset_code":"AS-1283","asset_label":"Concourse digital display network","asset_category":"signage","wording":"endeavour","precondition":"on_shortfall","cadence":"term_period","occurrence_basis":"not_applicable","standing":"remedy_only","window":"across_term"},{"asset_code":"AS-5648","asset_label":"Category exclusivity, domestic appliances","asset_category":"category_exclusivity","wording":"endeavour","precondition":"none_stated","cadence":"term_period","occurrence_basis":"not_applicable","standing":"not_determined","window":"across_term"},{"asset_code":"AS-3773","asset_label":"Pre-match interview backdrop placement","asset_category":"broadcast","wording":"not_stated","precondition":"none_stated","cadence":"per_occurrence","occurrence_basis":"schedule_referenced","standing":"not_determined","window":"not_determined"},{"asset_code":"AS-2371","asset_label":"Primary mark licence for sponsor advertising","asset_category":"ip_licence","wording":"grant","precondition":"none_stated","cadence":"single_milestone","occurrence_basis":"not_applicable","standing":"owed_this_term","window":"one_off"},{"asset_code":"AS-3819","asset_label":"App home-screen placement panel","asset_category":"digital","wording":"not_stated","precondition":"none_stated","cadence":"single_milestone","occurrence_basis":"not_applicable","standing":"not_determined","window":"one_off"},{"asset_code":"AS-8297","asset_label":"Away-fixture travel allocation, main party","asset_category":"hospitality","wording":"not_stated","precondition":"renewal_only","cadence":"per_occurrence","occurrence_basis":"schedule_referenced","standing":"not_owed_this_term","window":"not_determined"},{"asset_code":"AS-4153","asset_label":"Owned-channel post series, match previews","asset_category":"digital","wording":"grant","precondition":"none_stated","cadence":"term_period","occurrence_basis":"not_applicable","standing":"owed_this_term","window":"across_term"},{"asset_code":"AS-3556","asset_label":"Tunnel-mouth backdrop branding","asset_category":"signage","wording":"grant","precondition":"none_stated","cadence":"per_occurrence","occurrence_basis":"count_stated","standing":"owed_this_term","window":"each_occurrence"}]}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Turn a sponsorship agreement into a deliverables worksheet — 50 sponsorship agreements. One model answered, and every answer was then graded Six different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersSix ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The tracker floor (evals/baseline.py) and the scorer (evals/judge.py) are both pure code and cost $0.00 to run against any result set. The figure above is the ONE scored run's own token counts (50 agreements) priced at Google Gemini 3 Flash's published rate. The 6-pack calibration, the worked-example capture, the four prompt-split calls and the one live UI screenshot are separate calls and are not in it.

Cost driversWhat actually moves the bill

• The fixed prefix — system prompt (904 tokens), worksheet rulebook (1283) and field schema (1017) — is 3204 of the 4184 input tokens on the worked example, 76.6 pct, and it is byte-identical on every call. Nothing here caches it.
• Provider-side reasoning. 181223 of the run's 209384 output tokens (86.6 pct) were reasoning rather than the JSON worksheet — about 563 tokens of actual content per agreement. Reasoning was left at the provider's default and the experiment to turn it off has NOT been fired.
• Output length scales with the AGREEMENT, not with a fixed record — this kit's reply is a LIST, one object of nine fields per scheduled asset, so a five-asset pack and an eight-asset pack are different bills. The largest reply in the run was 14,432 output tokens against a mean of 4188.

Your volumeWhat it costs at your volume

Linear in agreement packs: each call is independent, carries the same fixed prefix and shares nothing with its neighbours, so 500 agreements cost ten times 50. Nothing amortises — there is no index and no cache. ⚠︎ AND IT IS NOT LINEAR IN THE LENGTH OF AN AGREEMENT. Output scales with the number of scheduled assets, and the ceiling is real rather than theoretical: the largest reply in this run used 14,432 of the 16,000 cap, 90.2 pct. A rights schedule longer than the eight assets this corpus builds needs the cap raised before the run, and the calibration will not tell you by how much — a six-pack subset saw 9,868.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 sponsorship agreement packs, one completion call each, one tier. The 6-pack calibration, the worked-example capture, the four prompt-split calls and the one live UI screenshot are recorded separately and are not in these totals.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the agreement pack into addressable sections, pure code — one per scheduled asset, plus the rights schedule, the notes, the fee reference and any decoy section

# Cut an agreement pack into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

map each field to the sections that can answer it, by exact name or heading prefix, and send their union. The Fee And Payment Reference section is mapped by nothing and therefore never leaves the machine; every decoy section is sent on purpose

You change it to: which sections answer which field, and therefore what is sent and what is not. This is the seam that WILL need editing for a different pack layout; when it matches nothing, selection falls back to the whole document — slower, more expensive, always correct.

# Pick which sections of an agreement pack are sent. Pure code -- the last deterministic step

AGREEMENT = "Agreement"
SCHEDULE = "Rights Schedule"
NOTES = "Agreement Notes"
SUPPLIED = "Sponsor-Supplied Items"
ANNEX = "Exclusivity Annex"
ASSET_PREFIX = "Asset "
NEVER_SENT = ("Fee And Payment Reference",)
SECTION_HINTS = {
def _matches(name, want):

src/rulebook.pyrulebook

load data/rulebook.json and turn four stated facts into two worksheet calls. Pure code, no model. The same module writes gold, renders the rule into the prompt, and re-runs it over the reply for the consistency diagnostic

# THE DELIVERABLE-WORKSHEET RULEBOOK, loaded from data/rulebook.json. Pure code, no model, no

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
WORDINGS = ("grant", "endeavour", "not_stated")
PRECONDITIONS = ("renewal_only", "on_shortfall", "none_stated")
CADENCES = ("per_occurrence", "term_period", "single_milestone", "not_stated")
OCCURRENCE_BASES = ("count_stated", "schedule_referenced", "not_applicable", "not_stated")
STANDINGS = ("owed_this_term", "not_owed_this_term", "remedy_only", "not_determined")
WINDOWS = ("each_occurrence", "across_term", "one_off", "not_determined")
def load(path=RULEBOOK_PATH):

src/prompt.pyprompt

assemble one prompt per agreement — system, the rulebook RENDERED from its JSON, the field schema, the selected sections — and parse the list-shaped reply, failing closed

# Assemble the deliverable-worksheet prompt. ONE prompt per agreement pack, every scheduled asset

SYSTEM = (
RULEBOOK_TEXT = rulebook_block()
def field_schema(contract_fields, item_fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

one model call per agreement, spans located back into each asset's own section, and compute() — the pure-code business flag — run over whatever came back

You change it to: the business condition the review flag fires on. It reads three enums off the reply and nothing else, so a different rights-holder's threshold is a different function body and no other change.

# Extract one agreement pack's deliverable worksheet: segment, select, prompt, one model call, then

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 16000
def load_fields():
def item_fields(fields=None):
def contract_fields(fields=None):
def load_doc(agreement_id):
def documents():

evals/judge.pyjudge

score it: identification as a set comparison over asset codes, cells by exact match, the two calls on matched rows only, the determinacy split, and the booked-as-owed count

# Score a deliverable-worksheet run. PURE CODE -- gold is exact and every answer is one value, so

def norm(v):
def code_norm(v):
def equal(got, want):
def _rows_of(rec):
def score_identification(records, golds):
CELL_FIELDS = ("asset_label", "asset_category", "wording", "precondition", "cadence",
def score(fields, records, golds):
CALLS = {"standing": RB.STANDINGS, "window": RB.WINDOWS}
CONFIDENT = {"owed_this_term", "not_owed_this_term", "remedy_only",

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3893 input and 4187 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's agreement packs are entirely synthetic (tools/build_corpus.py, seed 20260861): no real property, team, league, federation, venue, event, sponsor, brand, product, agency or agreement exists in the corpus, and nothing was fetched from anywhere. No governing body's regulation, league or federation rule, industry standard, measurement methodology or rate card is reproduced, paraphrased or named. The kit makes exactly one outbound request per agreement — the completion — and writes nothing anywhere: no database, no queue, no webhook, no file outside results/.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured provider cannot echo a credential onto the page. A shared per-day call ledger (src/budget.py) records only the day, a timestamp, the kit and the model — never a key and never a prompt.

The experimentWe did not attack it — and three of four boundaries hold

Three boundaries hold. One was confirmed by reading the code: no code path records fulfilment, awards a make-good or bills anybody, and the rulebook lookup and the review rule read only enums, so no clause text can reach either. One was confirmed by measurement: 0 phantom deliverables over 322 scheduled assets and 42 decoy codes, against a free floor that picks up 29 of them, so the opportunity was real. One was confirmed by two pre-flight assertions: the Fee And Payment Reference section reaches the model on none of the 50 packs, and no invented sponsor or property name appears anywhere in what does. The fourth — whether a clause written to talk a model out of the rulebook can move a call — is open, and no number is offered for it. Confirmed by reading the code, by measurement, and by two pre-flight assertions — not by an attack run — on 2026-08-22 — no attack run was fired; see redteam.why.

The first three boundaries hold, the third confirmed by two assertions that run before every scored run rather than by reading the code alone. The fourth is open and is named as open — an agreement is a document the other side helped write, and the Agreement Notes section is exactly where an instruction would arrive.

The result0 attack trials, and three of four boundaries checked here hold on evidence — one of them by two assertions that run before every scored run, not by reading the code. The one that does not is the one this kit's shape invites: an agreement is a document the other side helped write, and nothing here has tested whether a sentence in it can move a call.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260861), so no text in it came from an outside party and there was nothing adversarial to resist. The planted decoys measure whether a model reads the RIGHTS SCHEDULE and the section headings to decide what is scheduled when the row shape points the other way, and whether it declines a call the agreement does not settle. Neither measures whether it obeys an instruction hidden in the same pack. Those are different failures and only the first two are measured here.

HonestyWhat this does not prove

• Whether an indirect prompt injection in an Agreement Notes section could move a standing or window call. No attack run was fired, and the section is sent on every pack.
• Whether the kit behaves safely against a hostile provider — a response body crafted to break the parser or to smuggle text into the worksheet. parse() fails closed to {} on anything it cannot read, but nothing here fired a malicious body at it.
• Whether a real agreement pack would carry anything sensitive this kit mishandles beyond the fee section it already drops. A real pack may carry personal data in a hospitality allocation, and no field asks for it — but no field asked for the customer name in a sibling kit either, and it took a selection rule to keep it off the wire.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_sponsor_query fires when the pack carries at least one asset the property OWES THIS TERM — standing == "owed_this_term" — whose cadence is "per_occurrence" and whose occurrence_basis is anything other than "count_stated". All three values come out of the same reply; nothing else is read, and no clause text reaches the rule.

src/extract.py::compute(), called by extract() on every agreement, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by identical code. The rulebook lookup it reads lives in src/rulebook.py, and tools/build_corpus.py keeps a tiny local copy of the condition ONLY so the generator does not import the extractor — evals/check_labels.py asserts the two agree on every gold row before any run may spend.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a determination that anything was met or missed, a fulfilment record, a proof-of-performance check, a valuation, a shortfall calculation, a make-good entitlement or an invoice. Nothing in this kit counts what was delivered or bills anybody.
• It is NOT a check on whether the extracted facts are right. If the model misreads the cadence or the occurrence basis, the flag inherits the misreading — which is exactly what happened once, on SPN-0022.
• It is NOT any rights-holder's review policy. An unbounded repeating obligation is this kit's own simplification; no governing body's regulation, league or federation rule, industry standard, measurement methodology or inventory policy was consulted, and none is reproduced.
• It is NOT a completeness check on the agreement. A pack missing its master agreement, a side letter or a renewal notice looks exactly like a complete one to this rule.

WatchedWhat is watched, and why that one

The kit measures 16 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 9 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 50, extraction_cells 2626, failures 0, lines 322, refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

extraction · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on documents, extraction_cells, lines, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

extraction · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 7 chips that all say so.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/extract.py, on any change to data/rulebook.json (which moves the calls the flag reads), on any change to data/fields.json's allowed values for standing, cadence or occurrence_basis, and on any provider or model change. The measured selection table in compute()'s docstring is re-derived from gold, so a corpus change invalidates it too.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the rulebook is a JSON file read with json, the provider is reached over urllib, and the UI is one http.server handler and 240 lines of framework-free JavaScript. A forker clones it and runs it; there is no install step before anything works.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per agreement — segment, select, prompt, one call, parse, compute — with no branch, no loop and no agent. The only structural difference from a flat-record sibling is that the REPLY is a list, which is why identification is scored separately from the calls made about what was identified, and why the output ceiling is a live risk rather than a formality.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain — the JSON extraction from a fenced reply, the retry and backoff, the transport-failure branch, the span location, the confusion matrices. Each is small and each is a place a bug can live.
• No framework means no framework's ecosystem — no tracing, no eval harness beyond this one, no structured-output enforcement. src/prompt.py::parse fails closed to {} rather than to a plausible empty worksheet, because on this kit an empty list is a real answer.
• The provider abstraction covers exactly two wire formats. A third provider is one function and one entry in PROVIDERS, and until somebody writes it the kit does not run there.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-sponsor-deliverables on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b000-sponsor-deliverables-tracker, t000-sponsor-deliverables-stub recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Every sponsor name, property name, agreement reference, asset code, commercial category, quantity and fee is invented; no real property, team, league, federation, venue, event, sponsor, brand, product or agency is named anywhere, and no governing body's regulation, industry standard, measurement methodology or rate card is reproduced, paraphrased or named. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineWhich assets reached the worksheet

Did the run list exactly the assets this rights schedule schedules? Micro precision and recall over asset codes, with the two directions named and never averaged: a MISSED deliverable is a scheduled asset left off, a PHANTOM deliverable is a row the property does not owe. Phantoms are classified by which decoy produced them, because 'it listed the struck row', 'it listed what the sponsor supplies' and 'it invented a code' are different defects with different fixes.

MethodHow the test was run

evals/judge.py::score_identification, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, which records the scheduled assets AND the decoy codes for every pack; evals/check_labels.py asserts before any run may spend that every decoy code is in the pack, collides with no scheduled row, and that each of the three decoy kinds clears its floor.

How tight can the band be? No tolerance band — a code is on the worksheet or it is not, compared after whitespace stripping and upper-casing.

Cadence: Re-score on any change to data/rulebook.json's what_is_a_deliverable_here, to src/prompt.py's rule 1, or to the corpus's decoy composition.

The decisionWhen to reach for it

In one linePer-(asset, field) exact match, light normalisation

Every one of the eight per-asset fields plus one agreement id per pack, compared against gold by string equality after trimming, collapsing whitespace and lower-casing. THE DENOMINATOR IS GOLD'S ROWS, NOT THE RUN'S — a scheduled asset the run never returned scores as a miss on all eight of its cells rather than disappearing, so a run cannot raise this number by returning less.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, one row per pack. Every stated fact was written into the agreement text by the generator that recorded it, and check_labels.py asserts each is readable off the pack it labels.

How tight can the band be? No tolerance band. Values are drawn from closed vocabularies except the two string fields, and those are copied verbatim from the pack.

Cadence: Re-score on any change to data/fields.json, to the corpus's sentence templates, or to the provider.

The decisionWhen to reach for it

In one lineThe two rulebook calls, four-way each

standing — owed_this_term / not_owed_this_term / remedy_only / not_determined — and window — each_occurrence / across_term / one_off / not_determined — on the assets BOTH sides list. Matched rows only, deliberately: a call about an asset the run never found is an identification failure and is already counted once there.

MethodHow the test was run

evals/judge.py::score_calls, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The shipped rulebook's own lookup (src/rulebook.py) over the four facts the generator wrote into the pack. check_labels.py re-derives every gold call and fails if any disagrees; it was red-proven by flipping one and watching the gate convict it by name.

How tight can the band be? No tolerance band; both are exact-match over closed vocabularies.

Cadence: Re-score on any change to data/rulebook.json, to src/prompt.py's rules 3 to 5, or to the corpus's bucket composition.

The decisionWhen to reach for it

In one line⚑ The headline — assets recorded as owed that the property does not owe

Of every asset gold says the property does NOT owe this term — not vested, a remedy only, or never actually promised — how many did the run record as owed_this_term? This is the specific error a sponsorship worksheet exists to prevent, and it hides inside a plain accuracy figure: standing has four values and two of them are confident NOT-owed answers, so an over-confidence count alone cannot see it.

MethodHow the test was run

evals/judge.py::score_calls, the booked_as_owed block. In-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: gold's own standing call, which is the shipped rulebook's lookup over the four facts the pack states.

How tight can the band be? No tolerance band — the denominator is every gold row whose standing is not owed_this_term, scored over matched rows only.

Cadence: Re-score on any change to the standing rule's step order, which is the thing that makes renewal and make-good beat the verb.

The decisionWhen to reach for it

In one lineHow often it says the agreement does not settle it

Of the calls the agreement does NOT settle, how many did the run answer with a confident value anyway (overconfident), and of the calls it DOES settle, how many did the run decline (overcautious)? The two are counted separately and never netted, because the costs are different and do not cancel.

MethodHow the test was run

evals/judge.py::score_calls, the determinacy block. In-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: gold's not_determined rows — 225 of the 644 calls in this corpus, 100 on standing and 125 on the window.

How tight can the band be? Rates are per-direction, over their own denominators — 225 open calls and 419 settled ones. They are never averaged into one number.

Cadence: Re-score on any change to the rulebook, the corpus composition or the provider.

The decisionWhen to reach for it

In one lineneeds_sponsor_query against the same rule run over gold

Does the pure-code business flag fire on exactly the agreements where the condition holds? A full confusion matrix, with unanswered kept out of the negatives — folding an unanswered pack into 'true negative' would let a run that returns nothing score as a careful one.

MethodHow the test was run

evals/judge.py::score_flag, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute() run over GOLD's own asset rows. Same function, two inputs — it is a business condition, so it needs labels, unlike the consistency diagnostic beside it.

How tight can the band be? Binary per agreement; None (no readable row) is neither positive nor negative and is counted as unanswered.

Cadence: Re-check on any change to compute(), to data/rulebook.json (which moves the calls the flag reads), or to data/fields.json's allowed values for standing, cadence or occurrence_basis.

The decisionWhen to reach for it