The business caseThe problem this solves

A signed subscription contract does not arrive as a list of promises. It arrives as an order form and a stack of clauses, and somebody has to turn it into a worksheet: which lines are actually ordered, which of them the customer could take on its own, and whether each is supplied over a period or at a single moment. Three things on the page look like evidence and are not — a line's own price, which says what the customer is charged and nothing about separability; a rate card for work nobody ordered; and a clause for a line an amendment struck before signature, still printed in full. Someone opening each signed contract pack, listing the lines the order form actually orders, and then working out, line by line, two things the paperwork may or may not answer: can the customer take this line on its own, and is it supplied over a period or at a single moment. It is minutes per contract and it is every contract, and the part that goes wrong is not the listing: it is treating a line's own price as evidence that the customer could have bought it separately, and quietly recording a settled answer where the contract says nothing at all.

The inputThe actual contract packs

The corpus is 50 contract packs, 0.12 MB (txt 50). Plain text, one format, invented rather than fetched — a real subscription contract names a real customer, a real vendor's product and edition, real money and terms both sides consider commercially confidential, and there is no public corpus of (contract, correct worksheet) pairs for the same reason there is no public corpus of bank statements. Generating it also makes the label mechanical: gold's two calls are the shipped rulebook's own lookup run over the same three facts the pack states, never somebody's reading of what the deal really is.

Contract
--------
OBX-0001

Customer Reference
------------------
Larkfield Retail Group (commercial), account AC-46522, billing contact BC-7297.

Order Form
----------
Order form OF-66515, signed 2026-01-14. Currency USD. Lines as ordered:

  PO-7609   Platform subscription, professional edition          USD 115,000
  PO-9363   Implementation and configuration                     fee not separately stated
  PO-3771   Regional data-residency module                       USD 42,000
  PO-9286   Administrator training, 4 days                       USD 22,000

Item PO-7609
------------
Description: Platform subscription, professional edition
Type: subscription_platform
Fee on the order form: USD 115,000
Provided continuously across the 12-month initial term.

Item PO-9363
------------
Description: Implementation and configuration
Type: implementation_services
Fee on the order form: fee not separately stated
The platform is not made available to the customer's named users until the implementation acceptance certificate is signed.
Delivered across the 48-month period beginning on the service start date.

Item PO-3771
------------
Description: Regional data-residency module
Type: optional_module
Fee on the order form: USD 42,000
Provided continuously across the 24-month initial term.

Item PO-9286
------------
Description: Administrator training, 4 days
Type: training_days
Fee on the order form: USD 22,000
Complete when the final scheduled day is delivered and signed off.

Contract Notes
--------------
This pack is a draft for review.
Circulated for review only. The controller has not seen this pack and no determination has been made on any line in it.

The outcomeWhat a good result looks like

A worksheet row per ordered line — the line code, its description, its type, the three facts the pack states about it, and the two calls the shipped rulebook makes from them — plus one pure-code review flag taken from three of those values. Every call carries the rulebook's own reasoning and not_determined is a first-class answer. Nothing here determines an obligation, allocates a transaction price, concludes on timing, opens a schedule or writes an entry.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Turning a month of signed contract packs into worksheets before a controller opens them — the one tier measured here — 275 of 275 separation calls and 268 of 275 patterns, with 7 over-confident calls of 175
  100 pct on the separation call against the free price-line floor's 40.0 pct, and 4.0 pct over-confidence against the floor's 100 pct. It also left every one of the 39 decoy codes off, against the floor's 13 phantom promises.
• Deciding whether to trust the not_determined column at all — the column, with the split in front of you
  The two calls behave completely differently and averaging them would hide it. Separation: 105 of 105 open calls correctly reported open, 0 over-confident. Pattern: 63 of 70, 7 over-confident. And 0 of 375 settled calls were wrongly declined, on either — so the caution is not being bought with noise.

And where nothing here is good enough:

• Deciding whether a cheaper or a more deliberate model would do better here — nothing — this page cannot answer it
  Only one model was run. There is no tier comparison, no repeatability pair and no ablation on this page, and the separation grader is already at 275 of 275, so it could not rank two models even if both were fired. The delivery pattern, at 268 of 275, is the only grader here with room to move.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-obligation-extract. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts, the 50-pack picker populates, the worksheet draws its six column headings and the shipped rulebook renders in full from data/rulebook.json; clicking Build worksheet returns the no-key sentence rather than a stack trace, and the review row reads “not computed — the reply carried no line the rule could read” rather than “no”. python -m evals.check_labels passes with no network access at all, and so does the free floor.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per contract pack

Current processWhat it replaces

Someone opening each signed contract pack, listing the lines the order form actually orders, and then working out, line by line, two things the paperwork may or may not answer: can the customer take this line on its own, and is it supplied over a period or at a single moment. It is minutes per contract and it is every contract, and the part that goes wrong is not the listing: it is treating a line's own price as evidence that the customer could have bought it separately, and quietly recording a settled answer where the contract says nothing at all.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings; a pack with none falls back to one whole-document segment

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 50 packs cut into 514 sections in about two thousandths of a second, in process, with no model and no network. 464 of those sections are sent; the 50 that are not are exactly the Customer Reference sections, which carry the customer's name, its segment, an account number and a billing-contact reference and which no field asks for.

LicenceLicence

MIT — this repository's own licence. Every customer name, order-form reference, line code, fee and clause is invented; no real company, supplier, product, brand, person or contract is named, and no accounting standard, standard-setter or audit firm is named either. The worksheet rulebook this kit ships (data/rulebook.json) is ILLUSTRATIVE and is not an authority. It was written for this kit and reproduces no accounting standard, no standard-setter's guidance, no audit firm's manual and no company's own revenue policy.

Bring your ownBring your own contract packs

REPLACE data/rulebook.json FIRST. It is this kit's own construction, it reproduces no accounting standard, and everything downstream reads it: src/rulebook.py loads it, src/prompt.py renders it into every call, and tools/build_corpus.py writes gold with it. Then replace data/corpus/*.txt, write your own data/fields.json, and supply a gold row per pack carrying its ordered lines AND its decoys — the decoy list is what makes the phantom-obligation grader possible. SECTION_HINTS in src/select.py maps fields to section headings and will need editing for a different pack layout; when it does not match, selection falls back to the whole document — slower, more expensive, always correct. compute() in src/extract.py is this kit's own review rule and is the second thing to replace.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 3,067 tokens assembled, 1,568 are contexts — 51% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on OBX-0004 — the same pack the worked example above was captured on, so the four parts sum to exactly the 3067 input tokens that call actually reported. Four calls at max_tokens=1, each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. kits/UC0055-obligation-extract/results/tokens-p001-obligation-extract.json. ⛑ THE FIXED PREFIX IS 2346 of 3067 TOKENS (76.5 pct) and is paid on every one of the 50 calls.

Full promptThe prompt in full

You read a SUBSCRIPTION CONTRACT PACK -- an order form plus the clauses behind its lines -- and return a REVIEWER'S WORKSHEET of the promises in it. You return JSON and nothing else.

You are preparing a worksheet for a qualified reviewer. You never determine a performance obligation, never allocate a price, never conclude on timing, never open a revenue schedule and never write a journal entry. Your job is to say what the paperwork states, apply the rulebook given below, and NAME THE LINES THE PAPERWORK DOES NOT SETTLE.

RULES, in order of importance:
1. ONE OBJECT PER ORDERED LINE OF THIS ORDER FORM, AND NO OTHERS. A line struck by an amendment is not ordered, even though its clause is still printed in the pack. A rate card, day rate or price list for work nobody has ordered is not a line. An item supplied under an EARLIER order form and continuing is not a line. All three carry codes in the same format as a real line, and putting one on the worksheet puts a promise in front of a reviewer that does not exist.
2. REPORT WHAT THE PACK STATES. `charge`, `dependency` and `timing` are readings of the paperwork, not judgements about it. If a line's Item section says nothing about how it relates to the other lines, `dependency` is 'silent' -- that is a finding about the contract, and it is the most common one. Never infer a dependency from the line type, from the fee, or from what you know about how software is usually sold.
3. `separation` IS DECIDED ONLY BY THE RULEBOOK BELOW, from `charge` and `dependency`. Work through the four steps IN ORDER and STOP at the first that fires:
   a. dependency 'required_first' -> 'bundled'.
   b. dependency 'separately_available' -> 'distinct'. This holds WHATEVER the fee column says, including a line the order form prices at nothing.
   c. dependency 'silent' AND charge 'no_separate_charge' -> 'bundled'.
   d. anything else -> 'not_determined'.
4. A FEE OF ITS OWN IS A PRICE, NOT A STATEMENT ABOUT SEPARABILITY. A priced line whose clause says nothing about whether the customer could take it alone is 'not_determined'. It is not 'distinct'. This is the step a confident reader skips, and on this rulebook it is the wrong answer rather than a defensible one.
5. `pattern` IS DECIDED ONLY FROM `timing`: 'period' -> 'over_time', 'event' -> 'point_in_time', 'silent' -> 'not_determined'. It is asked of every line independently of its separation call.
6. 'not_determined' IS A REAL ANSWER AND YOU ARE EXPECTED TO USE IT. A worksheet that never reaches for it is guessing, and a call recorded as settled is a call nobody re-reads.
7. Copy `item_code`, `item_label` and `item_type` verbatim from the pack. Use the exact allowed value for every field that lists them, and return every field for every line.

WORKSHEET RULEBOOK (the authority for `separation` and `pattern`; this is an ILLUSTRATIVE rulebook written for this kit, and it reproduces no accounting standard)

WHAT COUNTS AS A LINE ON THE WORKSHEET
  - An item is on the worksheet if and only if it is a LINE OF THIS ORDER FORM that the contract supplies.
  - A line struck from the order form by an amendment is NOT on the worksheet, even though its code and its description are printed on the page.
  - A rate card, day rate or price list for work that has not been ordered is NOT on the worksheet. A price is not a promise.
  - An item supplied under an EARLIER order form and continuing is NOT on the worksheet. It is somebody else's paperwork, referred to here for context.

THE THREE STATED FACTS, read off the contract
  charge:
    separate_fee           the order form gives this line a fee of its own
    no_separate_charge     the order form states the line is included at no separate charge
    not_stated             the order form's fee column for this line says the fee is not separately stated
  dependency:
    required_first         the contract states this item must be completed, accepted or supplied before another ordered item can be used - or that it cannot be supplied except together with another ordered item
    separately_available   the contract states the customer may take, cancel, defer or obtain this item on its own, or from another supplier, without affecting the remaining items
    silent                 the contract says neither of those things about this item
  timing:
    period                 the contract states a period, term or window over which this item is supplied
    event                  the contract states a single completion, delivery, issue or acceptance event for this item
    silent                 the contract states neither

SEPARATION -- work through IN ORDER, stop at the first that fires
  1. DEPENDENCY STATED AS A PREREQUISITE. If dependency is `required_first`, the worksheet records the item as `bundled`. The paperwork itself ties it to another ordered item, so this rulebook does not carry it as a promise on its own.
  2. DEPENDENCY STATED AS SEPARABLE. If dependency is `separately_available`, the worksheet records the item as `distinct`. This is true whatever the fee line says, including a line the order form prices at nothing.
  3. NOTHING STATED, AND THE LINE IS EXPLICITLY FREE. If dependency is `silent` and charge is `no_separate_charge`, the worksheet records the item as `bundled`. An order form that gives a line no price of its own AND says nothing about taking it separately is, on this rulebook, recording it alongside the line it rides on.
  4. NOTHING STATED, ANYTHING ELSE. The worksheet records `not_determined`. A FEE OF ITS OWN IS A PRICE, NOT A STATEMENT ABOUT SEPARABILITY - it says what the customer is charged, not whether the customer could have taken the item alone. This is the branch a confident reader skips.

DELIVERY PATTERN
  1. If timing is `period`, the worksheet records `over_time`.
  2. If timing is `event`, the worksheet records `point_in_time`.
  3. If timing is `silent`, the worksheet records `not_determined`. The pattern question is asked of every item independently of the separation call - a bundled item still either has a stated window, a stated event, or neither.

WHY `not_determined` IS A REAL ANSWER
  A worksheet that never says `not_determined` is not a confident worksheet, it is a guessing one. The whole value of this shape is that a reviewer can see, in one column, which calls the paperwork actually settles and which ones somebody has to go back to the deal desk about. Over-confidence here costs more than caution: a call recorded as settled is a call nobody re-reads.

Return these:
- contract_id (string) -- the contract identifier from the Contract section, verbatim
- obligations (array of objects) -- one object per ORDERED line, each carrying:
    - item_code (string) -- the line code from the Order Form row, verbatim (for example PO-4417). THIS IS THE KEY -- one object per ORDERED line and no others
    - item_label (string) -- the line's description, copied verbatim from its own Item section's Description line
    - item_type (enum) one of: subscription_platform, implementation_services, training_days, support_tier, usage_component, renewal_option, optional_module -- the line type, copied verbatim from the Type line of the line's own Item section
    - charge (enum) one of: separate_fee, no_separate_charge, not_stated -- what the ORDER FORM's fee column says about this line. A money amount is 'separate_fee'; the words 'included at no separate charge' are 'no_separate_charge'; the words 'fee not separately stated' are 'not_stated'. Report what the paperwork says -- this is NOT on its own what decides `separation`
    - dependency (enum) one of: required_first, separately_available, silent -- what the line's Item section STATES about how it relates to the other ordered lines. 'required_first' when it says this line must be completed, accepted or supplied before another ordered item can be used, or that it cannot be supplied except together with another ordered item. 'separately_available' when it says the customer may take, cancel, defer or obtain this line on its own or from another supplier without affecting the remaining lines. 'silent' when the section says NEITHER. Silence is the common case and it is a finding, not a gap -- never infer a dependency from the line type, from the fee, or from what you know about how software is usually sold
    - timing (enum) one of: period, event, silent -- what the line's Item section STATES about when it is supplied. 'period' when it names a period, term or window; 'event' when it names a single completion, delivery, issue or acceptance event; 'silent' when it names neither
    - separation (enum) one of: distinct, bundled, not_determined -- the worksheet's separation call for this line, decided STRICTLY by the shipped rulebook from `charge` and `dependency` and nothing else. Work through the four steps IN ORDER, stopping at the first that fires. (1) dependency 'required_first' -> 'bundled'. (2) dependency 'separately_available' -> 'distinct', whatever the fee column says. (3) dependency 'silent' AND charge 'no_separate_charge' -> 'bundled'. (4) anything else -> 'not_determined'. A FEE OF ITS OWN IS A PRICE, NOT A STATEMENT ABOUT SEPARABILITY: a priced line with nothing said about taking it alone is 'not_determined', never 'distinct'
    - pattern (enum) one of: over_time, point_in_time, not_determined -- the worksheet's delivery-pattern call, decided STRICTLY from `timing`: 'period' -> 'over_time', 'event' -> 'point_in_time', 'silent' -> 'not_determined'. Asked of every line independently of its separation call -- a bundled line still either has a stated window, a stated event, or neither

Return a JSON object with exactly these top-level keys: contract_id, obligations
`obligations` is an array. Return it empty only if the order form has no ordered line at all.

CONTRACT PACK
-------------
Contract
--------
OBX-0004

Order Form
----------
Order form OF-74140, signed 2026-02-14. Currency USD. Lines as ordered:

  PO-6949   Platform subscription, standard edition              USD 192,000
  PO-2213   Implementation and configuration                     USD 57,000
  PO-2070   Regional data-residency module                       USD 15,000
  PO-6508   Extension right over the following term              included at no separate charge
  PO-2195   Administrator training, 5 days                       USD 17,000
  PO-6476   Metered API calls above the included allowance       included at no separate charge
  PO-8102   Analytics module                                     WITHDRAWN by amendment A-1 -- not supplied under this order form

Item PO-6949
------------
Description: Platform subscription, standard edition
Type: subscription_platform
Fee on the order form: USD 192,000
The customer may cancel this line at any time without affecting the remaining lines of this order form.
Delivered across the 12-month period beginning on the service start date.

Item PO-2213
------------
Description: Implementation and configuration
Type: implementation_services
Fee on the order form: USD 57,000
No part of the subscription may be used until this work is accepted in writing.
Complete on the date the access credentials are issued to the customer.

Item PO-2070
------------
Description: Regional data-residency module
Type: optional_module
Fee on the order form: USD 15,000
Complete on the date the access credentials are issued to the customer.

Item PO-6508
------------
Description: Extension right over the following term
Type: renewal_option
Fee on the order form: included at no separate charge
Complete on signature of the acceptance certificate.

Item PO-2195
------------
Description: Administrator training, 5 days
Type: training_days
Fee on the order form: USD 17,000

Item PO-6476
------------
Description: Metered API calls above the included allowance
Type: usage_component
Fee on the order form: included at no separate charge

Item PO-8102
------------
Description: Analytics module
Type: optional_module
Fee on the order form: withdrawn
This clause was drafted before amendment A-1 and is retained in the pack for the audit trail.

Professional Services Rate Card
-------------------------------
Rate card reference PO-7231, consultancy day rate USD 2,180 per day.
No services are ordered under this order form. These rates apply only to work ordered later under a separate statement of work.

Continuing Items From An Earlier Order Form
-------------------------------------------
Item PO-3726, Extended-hours support tier, was supplied under order form OF-43294 and continues in effect.
It is not re-ordered here and is listed for context only.

Contract Notes
--------------
This pack is a draft for review.
Amendment A-1 removed line PO-8102 before signature; it is not supplied under this order form.
Circulated for review only. The controller has not seen this pack and no determination has been made on any line in it.

Raw responseThe raw response

{"contract_id":"OBX-0004","obligations":[{"item_code":"PO-6949","item_label":"Platform subscription, standard edition","item_type":"subscription_platform","charge":"separate_fee","dependency":"separately_available","timing":"period","separation":"distinct","pattern":"over_time"},{"item_code":"PO-2213","item_label":"Implementation and configuration","item_type":"implementation_services","charge":"separate_fee","dependency":"required_first","timing":"event","separation":"bundled","pattern":"point_in_time"},{"item_code":"PO-2070","item_label":"Regional data-residency module","item_type":"optional_module","charge":"separate_fee","dependency":"silent","timing":"event","separation":"not_determined","pattern":"point_in_time"},{"item_code":"PO-6508","item_label":"Extension right over the following term","item_type":"renewal_option","charge":"no_separate_charge","dependency":"silent","timing":"event","separation":"bundled","pattern":"point_in_time"},{"item_code":"PO-2195","item_label":"Administrator training, 5 days","item_type":"training_days","charge":"separate_fee","dependency":"silent","timing":"silent","separation":"not_determined","pattern":"not_determined"},{"item_code":"PO-6476","item_label":"Metered API calls above the included allowance","item_type":"usage_component","charge":"no_separate_charge","dependency":"silent","timing":"silent","separation":"bundled","pattern":"not_determined"}]}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch the priced lines a subscription contract leaves open — 50 contract packs. One model answered, and every answer was then graded Five different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersFive ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

The price-line floor (evals/baseline.py) and the scorer (evals/judge.py) are both pure code and cost $0.00 to run against any result set. The figure above is the ONE scored run's own token counts (50 packs) priced at Google Gemini 3 Flash's published rate — the same basis cost_per_query_usd uses. ⚠︎ IT IS NOT WHAT BUILDING THIS KIT COST. Four further paid passes sit outside it and are committed in results/: a 6-pack calibration that MEASURED max_tokens ($0.0426), the worked-example capture ($0.0187), 4 prompt-split calls at max_tokens=1 ($0.0039) and 2 live UI screenshots ($0.0373). All 63 live calls together project to $0.4830 on this card.

Cost driversWhat actually moves the bill

• The fixed prefix — system prompt (753 tokens), worksheet rulebook (847) and field schema (746) — is 2346 of 3067 tokens on the example call, 76.5 pct, and is byte-identical on every one of the 50 calls. The contract pack itself is 721 tokens. The bill is the rulebook and the instructions, not the contract.
• Provider-side reasoning. 82693 of the run's 102608 output tokens (80.6 pct) were reasoning rather than the JSON worksheet, on a reply that is about 400 tokens of actual content. Reasoning was left at the provider's default and nothing here has measured what turning it off would cost in accuracy.
• Output length scales with the PACK, not with a fixed record — this kit's reply is a LIST. A four-line pack and a seven-line pack are not the same call, and the run's largest reply was 6,776 output tokens against a smallest of well under half that.

Your volumeWhat it costs at your volume

Linear in contract packs: each call is independent, carries the same fixed prefix and shares nothing with its neighbours, so 500 packs cost ten times 50. Nothing amortises — there is no index and no cache. ⚠︎ AND IT IS NOT LINEAR IN LINES PER PACK, WHICH IS THE AXIS THAT ACTUALLY BREAKS. The reply is a list, and the run's longest was 85 pct of the published 8,000-token ceiling on a seven-line pack. A twenty-line pack does not fit under this cap at all, and the honest fix is a different shape — lines batched across calls — not a bigger number.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 subscription contract packs, one completion call each, one tier. The calibration, the worked-example capture, the prompt-split calls and the UI screenshots are recorded separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the contract pack into addressable sections, pure code — one per ordered line, plus the order form, the notes and any decoy clause

# Cut a document into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections are sent, pure code — section names are matched by PREFIX because every pack's Item sections are named after its own line codes. The two decoy sections are deliberately SENT; the Customer Reference section is mapped by nothing and never leaves the machine

You change it to: map fields to your own pack's headings, including the per-line prefix; unmatched falls back to the whole document. NEVER_SENT is the list of sections no field asks for and which therefore never leave the machine

# Pick which sections of a contract pack are sent. Pure code -- the last deterministic step

ORDER_FORM = "Order Form"
CONTRACT = "Contract"
NOTES = "Contract Notes"
RATE_CARD = "Professional Services Rate Card"
CARRYOVER = "Continuing Items From An Earlier Order Form"
ITEM_PREFIX = "Item "
NEVER_SENT = ("Customer Reference",)
SECTION_HINTS = {
def _matches(name, want):

src/rulebook.pyrulebook

the two worksheet calls, loaded from data/rulebook.json — a four-step separation rule with a stopping order and a three-way pattern rule. Data, not code, so a forker can open it and disagree with it

# THE WORKSHEET RULEBOOK, loaded from data/rulebook.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
CHARGES = ("separate_fee", "no_separate_charge", "not_stated")
DEPENDENCIES = ("required_first", "separately_available", "silent")
TIMINGS = ("period", "event", "silent")
SEPARATIONS = ("distinct", "bundled", "not_determined")
PATTERNS = ("over_time", "point_in_time", "not_determined")
def load(path=RULEBOOK_PATH):
R = load()

src/prompt.pyprompt

assemble one call per pack for every ordered line, with the rulebook RENDERED into it from data/rulebook.json rather than restated in prose — a rule the model has never been shown is not a rule

# Assemble the worksheet prompt. ONE prompt per contract pack, every ordered line in the reply.

SYSTEM = (
RULEBOOK_TEXT = rulebook_block()
def field_schema(contract_fields, item_fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key — plus the pure-code business-condition check downstream: a pack carrying a PRICED line whose separation and pattern are both open goes back to the deal desk

You change it to: the review rule itself — this kit ships three values (a fee of its own, and both calls open) and a real review weighs materiality, side letters and what the sales team knows. It is deliberately NOT the same function as the rulebook lookup, so changing WHAT GETS REVIEWED does not change WHAT THE RULEBOOK SAYS

# Extract one contract pack's worksheet: segment, select, prompt, one model call, then a pure-code

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 8000
def load_fields():
def item_fields(fields=None):
def contract_fields(fields=None):
def load_doc(contract_id):
def documents():

evals/judge.pyjudge

score identification, the extracted cells, the two calls and the determinacy split, all separately, pure code — plus a no-gold consistency diagnostic

# Score an obligation-worksheet run. PURE CODE -- gold is exact and every answer is one value, so

def norm(v):
def code_norm(v):
def equal(got, want):
def _rows_of(rec):
def score_identification(records, golds):
CELL_FIELDS = ("item_label", "item_type", "charge", "dependency", "timing",
def score(fields, records, golds):
CALLS = {"separation": RB.SEPARATIONS, "pattern": RB.PATTERNS}
def _confusion(rows):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2906 input and 2052 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's contract packs are entirely synthetic (tools/build_corpus.py, seed 20260822): no real customer, supplier, product, contract, order form or fee exists in the corpus, and nothing was fetched from anywhere. No accounting standard, standard-setter, audit firm or software vendor is named anywhere in the kit. The only outbound traffic the kit makes is one chat-completion request per pack to the configured provider, carrying the mapped sections of one pack plus the fixed prompt and the shipped rulebook — and the Customer Reference section, which is the one part of a real pack a deal desk would most mind sending anywhere, is mapped by no field and never leaves the machine. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser.

The experimentWe did not attack it — and three of four boundaries hold

Three boundaries hold. Two were confirmed by reading the code: no code path determines, allocates, schedules or posts anything, and the rulebook lookup and the review rule read only enums, so no clause text can reach either. The third is confirmed by a CHECK RATHER THAN BY READING — evals/check_labels.py asserts, before any run may spend, that the Customer Reference section is never sent and that every decoy section is. The fourth is open. An indirect prompt injection needs a section an outside party authored, and a contract is exactly that. Confirmed by reading the code and by a pre-flight assertion, not by an attack run, on 2026-08-22 — no attack run was fired; see redteam.why.

The first three boundaries hold, the third confirmed by an assertion that runs before every scored run rather than by reading the code alone. The fourth is open and is named as open — a contract is a document the other side helped write, which makes an instruction hidden in a clause exactly the attack this kit's shape invites, and nothing here has tried it.

The result0 attack trials, and three of four boundaries checked here hold on evidence — one of them by an assertion that runs before every scored run, not by reading the code. The one that does not is the one this kit's shape invites: a contract is a document the counterparty helped draft, and whether an instruction hidden in a clause could move a call is unmeasured.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted decoys measure whether a model reads the ORDER FORM to decide what is ordered when the clauses point the other way, and whether it declines a call the paperwork does not settle. Neither measures whether it obeys an instruction hidden in the same pack. Those are different failures and only the first two are measured here.

HonestyWhat this does not prove

• Whether an indirect prompt injection in a Contract Notes section could move a call. No attack run exists.
• Whether the kit behaves safely against a hostile provider — a response body crafted to break the list parsing in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed document, and nothing beyond that was tested.
• Whether a real contract pack would carry anything sensitive this kit mishandles beyond the customer section. The corpus has no personal data by construction; a real pack carries named signatories, negotiated pricing and terms both sides consider confidential, and none of that path is exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_drafting_review fires when the pack carries at least one line the order form PRICES — charge == "separate_fee" — whose separation AND delivery pattern are both "not_determined". All three values come out of the same reply; the rule is run afterwards in pure code, over whatever the model returned, and never over gold. A reply carrying no line the rule can read returns None rather than False — an unknown is not a pass.

src/extract.py::compute(), called by extract() on every pack, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by identical code. The rulebook lookup it reads lives in a SEPARATE module, src/rulebook.py, which is also what the corpus generator wrote gold with and what src/prompt.py states to the model — one definition, three readers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT an accounting conclusion, a determination, an allocation, a schedule or an entry. Nothing here determines a performance obligation, allocates a transaction price, concludes on timing, opens a revenue schedule, posts anything or notifies anybody. separation and pattern are proposals with the rulebook's own reasoning attached, and a controller decides.
• It is NOT a check on whether the extracted facts are right. If the model misreads a clause and then applies the rulebook correctly to the misreading, the pack is flagged — or not flagged — on a wrong fact, and this rule cannot tell. THAT IS NOT HYPOTHETICAL HERE: all 7 of the run's errors are exactly that shape, and the consistency diagnostic in evals/judge.py, which is the closest thing to that check, caught 0 of them.
• It is NOT any company's review policy. A priced line with both calls open is this kit's own simplification, invented for this corpus. No revenue policy, audit manual, accounting standard or standard-setter's guidance was consulted, and none is reproduced. A real review weighs the fee against materiality, whether a side letter exists, and what the sales team knows and did not write down.
• It is NOT a judgement about the rulebook. The flag reads the calls the rulebook produced; if the rulebook is wrong, the flag is confidently wrong with it. The shipped rulebook is illustrative, reproduces no accounting standard, and nobody qualified has reviewed it.

WatchedWhat is watched, and why that one

The kit measures 15 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 8 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 50, extraction_cells 1975, failures 0, lines 275, refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

extraction · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on documents, extraction_cells, lines, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

extraction · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 7 chips that all say so.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/extract.py, on any change to data/rulebook.json (which moves the calls the flag reads), on any change to data/fields.json's allowed values for charge, separation or pattern, and on any corpus regeneration. Changing WHAT GOES BACK TO THE DEAL DESK is a policy change and must be re-scored, even though it never changes what the rulebook says.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the rulebook is a JSON file read with json, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per pack — segment, select, prompt, one call, parse, compute — with no branch, no loop and no agent. The only structural difference from its siblings is that the REPLY is a list, which changes the scorer and not the pipeline.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain — the JSON extraction from a fenced reply, the list parsing, the backoff policy, the section regex, the prefix matcher and the .env reader are all code somebody has to own. This kit paid a small instalment of that during its own build: the stub's order-form regex cut the section at the first blank line, which sits between an order form's header sentence and its rows, so it reported fifty packs with zero lines each while every other part of the wiring was fine.
• No framework means no framework's ecosystem — no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. The reply is a LIST and nothing validates its shape except src/prompt.py::parse, which fails closed to {} rather than to an empty worksheet, because an empty obligations list is a legitimate answer here.
• The provider abstraction covers exactly two wire formats. A third provider is one function and one dict entry, and until somebody writes it the kit runs on two shapes.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-obligation-extract on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b000-obligation-extract-priceline, t000-obligation-extract-stub recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Every customer name, order-form reference, line code, fee and clause is invented; no real company, supplier, product, brand, person or contract is named, and no accounting standard, standard-setter or audit firm is named either. The worksheet rulebook this kit ships (data/rulebook.json) is ILLUSTRATIVE and is not an authority. It was written for this kit and reproduces no accounting standard, no standard-setter's guidance, no audit firm's manual and no company's own revenue policy. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineWhich lines reached the worksheet

Did the run list exactly the lines this order form orders? Micro precision and recall over line codes, with the two directions named and never averaged: a MISSED obligation is a real line left off, and a PHANTOM obligation is a line on the worksheet that is not one. Phantoms are broken down by which decoy produced them — an amendment-struck line, a rate card, an item continuing under an earlier order form, or a code that appears nowhere in the pack.

MethodHow the test was run

evals/judge.py::score_identification, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, which records the ordered lines AND the decoy codes for every pack; evals/check_labels.py asserts before any run may spend that every decoy code is in the pack, collides with no ordered line, and that each of the three kinds clears a floor of 8.

How tight can the band be? No tolerance band — a code is on the worksheet or it is not, compared after whitespace stripping and upper-casing.

Cadence: Re-score on any change to data/rulebook.json's what_is_an_obligation_here, to src/prompt.py's rule 1, or to the corpus's decoy composition.

The decisionWhen to reach for it

In one linePer-(line, field) exact match, light normalisation

Does the model's value for each of the seven per-line fields match gold, after trimming whitespace and punctuation, plus one contract id per pack? THE DENOMINATOR IS GOLD'S LINES, NOT THE RUN'S: a gold line the run never returned scores as a miss on all seven of its cells rather than disappearing, so a run cannot raise this number by returning less.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every run, and the same logic the free floor is scored by — a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same values the pack states; evals/check_labels.py asserts every field is populated and in its allowed set on every one of the 275 lines, and that every value is stated in the pack it labels, before any run is allowed to spend.

How tight can the band be? There is no tolerance band — it is exact match after trimming whitespace and punctuation, never a continuous score to round.

Cadence: Re-score on any change to data/rulebook.json, src/prompt.py, data/fields.json or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe two rulebook calls, three-way each

Did the run land on the same calls the shipped rulebook produces — distinct / bundled / not_determined for separation, and over_time / point_in_time / not_determined for the delivery pattern? SCORED ON MATCHED LINES ONLY, and saying so is half of what makes the number honest: a call about a line the run never found has already been counted, once, as an identification failure, and counting it again here would let one mistake sink two graders.

MethodHow the test was run

evals/judge.py::score_calls, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's calls are re-derived inside the grader by the same lookup the kit publishes — src/rulebook.py, reading data/rulebook.json — so the truth this grades against can never be a separately-typed label that drifted from the rule.

How tight can the band be? No threshold — each call is one of three allowed values, and a reply returning none of them is counted as unanswered rather than folded into a correct cell.

Cadence: Re-run whenever data/rulebook.json or src/rulebook.py changes, whenever the corpus is regenerated, and on any provider or model change.

The decisionWhen to reach for it

In one line⚑ The headline — how often it says the paperwork does not settle it

Of every call this corpus's contracts do NOT settle, how many did the run settle anyway? That is overconfident, and on a reviewer's worksheet it is the expensive direction: a call recorded as settled is a call nobody re-reads, so an over-confident worksheet quietly removes the reviewer it exists to serve. Its mirror, overcautious — declining a call the contract does settle — is counted and published beside it, never netted against it, because the two costs are different: one hides work, the other creates it.

MethodHow the test was run

evals/judge.py::score_calls — the determinacy block, in-process, no key, no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The same rulebook lookup as the grader above, with the rows partitioned by whether GOLD's answer is not_determined. Both directions come out of one pass over the same 550 calls.

How tight can the band be? A rate over a fixed denominator: 175 open calls, 375 settled ones. A run over a different corpus has a different denominator and the two rates are not comparable.

Cadence: Re-run on any change to the rulebook's not_determined branches, to the corpus's silent-line composition, or to the provider or model.

The decisionWhen to reach for it

In one lineneeds_drafting_review against the same rule run over gold

Does the pure-code review decision — a PRICED line whose separation and delivery pattern the paperwork settles neither of — land on the same packs it would land on if every line had been read perfectly? It is a BUSINESS CONDITION, so unlike a self-consistency check it genuinely needs labels, and saying so is half of what makes the number believable.

MethodHow the test was run

evals/judge.py::score_flag, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute(), the same function the run uses, applied to GOLD's own lines. One rule, one input shape, so a change to the rule moves both sides together and the grader cannot silently grade an old policy.

How tight can the band be? No threshold — three values per line and two comparisons.

Cadence: Re-run whenever compute() changes. Changing WHAT GOES BACK TO THE DEAL DESK is a policy change and must be re-scored, even though it never changes what the rulebook says.

The decisionWhen to reach for it