The business caseThe problem this solves

A payer-provider agreement states reimbursement terms in prose — a rate basis, an effective date, carve-outs, escalators, term-specific exceptions. Somebody then CONFIGURES those terms into a claims system, and the configuration extract is a structured record. Checking one against the other looks like a two-column diff and is not. Three things on the page look like evidence and are not: the SCHEDULE OF TERMS, which was cut at execution and is never re-cut, so it shows the pre-amendment basis; the configuration's own value, which is the thing under review and cannot be evidence about the contract; and the headline rate agreeing, which says nothing about an exclusion stated three paragraphs further down the same Article. Somebody opening an executed agreement and the configuration extract loaded from it, and working through them term by term: does the rate basis the claims system carries match the basis the Agreement operatively states, does it carry the exclusions the Article attaches to it, and is it tied to the same event. It is minutes per term and it is every term, on every amendment, on every renewal. The part that goes wrong is not the comparison: it is that the two fastest places to look — the schedule of terms and the configuration table — agree with each other on exactly the terms an amendment has moved.

The inputThe actual contract extracts

The corpus is 50 contract extracts, 0.33 MB (txt 50). Plain text, one format, invented rather than fetched — and this row was CHOSEN because it can be built without a single record about a person receiving care, which is what makes it publishable at all. A real payer-provider agreement names a real payer and a real provider group, carries rates both sides treat as the most sensitive numbers they hold, and sits under a confidentiality clause; its configuration extract comes out of a live claims system. There is no public corpus of (agreement, configuration, correct worksheet) triples. Generating it also makes the label mechanical: gold's two calls are the shipped rulebook's own lookup run over the same six facts the two documents state, never somebody's reading of what the deal really is.

RILLWATER HEALTH PLAN
PROVIDER CONTRACTING — CONTRACT AND CONFIGURATION REVIEW EXTRACT

SYNTHETIC RECORD. This is generated test paperwork. No member, patient or claim record of
any kind appears in it; the payer, the provider group, every code and every amount are
invented. It reproduces no statute, regulation, standard or real rate schedule.

Provider group:      Weatherby Regional Group
Agreement ref:       PC-2022-438
Extract id:          CLQ-0001
Configuration load:  CFG-2023-4901

This extract pairs the executed Agreement with the configuration record loaded from it for
review. The Schedule below is the execution snapshot; the Articles set out each term; any
Amendment follows them; the Configuration Extract at the foot is what the claims system
carries today.

CONTRACTING CONTACT AND NEGOTIATION RECORD

  Correspondence address:  24 Harrow Lane, Stonebridge
  Telephone:               01930 743228
  Network manager:         R. Halloran
  Last negotiation round:  2025-06
  Escalation preference:   written, to the network manager named above

SCHEDULE OF TERMS AT EXECUTION
--------------------------------------------------------------------------------------------
CODE        SERVICE CATEGORY                RATE BASIS AT EXECUTION   STATUS
--------------------------------------------------------------------------------------------
TRM-8219    Diagnostic imaging              percent of schedule       executed
TRM-3764    Long-term care                  not stated                executed
TRM-1270    Emergency services              percent of schedule       executed
--------------------------------------------------------------------------------------------

ARTICLE 1 — Diagnostic imaging (TRM-8219)

The outcomeWhat a good result looks like

A worksheet row per term — the term code, the service category, six stated facts (three read off the Agreement and three off the configuration), whether an amendment restated the term, the operative rate sentence verbatim, and the two calls the shipped rulebook makes — plus one pure-code review flag. Every call carries the rulebook's own reasoning and not_determined is a first-class answer. Nothing here determines payment, adjudicates a claim, authorises a reprocessing or changes a configuration.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Checking a newly loaded or newly amended agreement before the next claim cycle runs on it — the one tier measured here — 275 of 275 on both calls, with 0 over-confident calls of 150 and 18 of 18 review flags
  100 pct on the rate call against the free two-table floor's 42.91 pct, and 0 pct over-confidence against the floor's 100 pct. It caught all 22 terms configured to a rate basis an executed amendment replaced, which the floor calls clean, and all 45 terms where the headline matches and the carve-out does not, which the floor also calls clean.
• Deciding how much output-token headroom to give a reasoning model on this shape — far more than a calibration will tell you — this kit ships 48,000 for a workload whose median finished reply is about 6,000
  A 6-extract calibration saw a largest reply of 10,908 tokens. A full run at 24,000 — 2.2x that — still lost a document, at exactly the ceiling, with all 24,000 output tokens reported as reasoning and no JSON at all. The same document replayed at 48,000 finished in 8,989. The cap is not covering the size of the answer; it is covering the tail of the process that produces it.

And where nothing here is good enough:

• Deciding whether a cheaper or a more deliberate model would do better here — nothing — this page cannot answer it
  Only one model was run. There is no tier comparison, no repeatability pair and no ablation on this page, and all four graders are now at 100 pct, so this corpus could not rank two models even if both were fired. It has run out of room.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-contract-load-qa. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts on 127.0.0.1:8919, the 50-extract picker populates, the worksheet draws its six column headings and the shipped rulebook renders in full from data/rulebook.json; clicking Build worksheet returns the no-key sentence rather than a stack trace, and the review row reads “not computed — the reply carried no term the rule could read” rather than “no”. python -m evals.check_labels passes with no network access at all, and so does the free two-table floor.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per extract

Current processWhat it replaces

Somebody opening an executed agreement and the configuration extract loaded from it, and working through them term by term: does the rate basis the claims system carries match the basis the Agreement operatively states, does it carry the exclusions the Article attaches to it, and is it tied to the same event. It is minutes per term and it is every term, on every amendment, on every renewal. The part that goes wrong is not the comparison: it is that the two fastest places to look — the schedule of terms and the configuration table — agree with each other on exactly the terms an amendment has moved.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureEvalsEvalsPromptPromptPromptPromptArchitectureArchitectureApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 0 of 9 stations carry a recorded failure

scroll → 9 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on fixed headings plus one section per written-up term, named by TERM CODE rather than article number so an amendment that renumbers the articles cannot move a citation; an extract with no headings falls back to one whole-document segment

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 50 extracts cut into 581 sections in a few thousandths of a second, in process, with no model and no network. 498 of those sections are sent; of the 83 that are not, 50 are exactly the CONTRACTING CONTACT AND NEGOTIATION RECORD sections and 33 are END OF EXTRACT footers.

LicenceLicence

MIT — this repository's own licence. NO MEMBER DATA AND NO PATIENT DATA APPEARS ANYWHERE IN THIS KIT. Every document is contract and configuration paperwork generated from a fixed seed: an executed payer-provider agreement and the configuration record loaded from it. There is no member, no patient, no claim for a real person, no diagnosis, no clinical detail and no identifier belonging to anyone receiving care, and every payer, provider group, term code and amount is invented. evals/check_labels.py asserts it over every line of every file before any run may spend. No real procedure code, rate schedule or code scheme appears either, real or invented to look real. The contract-load rulebook this kit ships (data/rulebook.json) is ILLUSTRATIVE and is not an authority. It was written for this kit. It reproduces no statute, no regulation, no standard, no payer's configuration manual and no provider group's contracting policy, and it names none of them.

Bring your ownBring your own contract extracts

REPLACE data/rulebook.json FIRST. It is this kit's own construction, It reproduces no statute, no regulation, no standard, no payer's configuration manual and no provider group's contracting policy, and it names none of them. and everything downstream reads it: src/rulebook.py loads it, src/prompt.py renders it into every call, and tools/build_corpus.py writes gold with it. Then replace data/corpus/*.txt, write your own data/fields.json, and supply a gold row per extract carrying its terms AND its decoys — the decoy list is what makes the phantom-term grader possible. SECTION_HINTS in src/select.py maps fields to section headings and will need editing for a different layout; when it does not match, selection falls back to the whole document minus NEVER_SENT — slower, more expensive, always correct. compute() in src/extract.py is this kit's own review rule and is the second thing to replace.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 4,783 tokens assembled, 1,394 are rules — 29% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on CLQ-0021: four calls at max_tokens=1, each prefix the previous plus exactly one more part.

Full promptThe prompt in full

You read a CONTRACT AND CONFIGURATION REVIEW EXTRACT -- an executed payer-provider agreement and the configuration record loaded from it into a claims system -- and return a REVIEWER'S WORKSHEET naming the terms the configuration does not match or the agreement does not settle. You return JSON and nothing else.

You are preparing a worksheet for a qualified reviewer. You never determine payment, never adjudicate a claim, never authorise or instruct the reprocessing of anything and never change a configuration. Your job is to say what the two documents state, apply the rulebook given below, and NAME THE TERMS THE PAPERWORK DOES NOT SETTLE.

RULES, in order of importance:
1. ONE OBJECT PER TERM ON THE WORKSHEET, AND NO OTHERS. A term belongs on the worksheet if the Agreement's OPERATIVE Articles state it, or the configuration extract carries a row for it. A term in the PROPOSED TERMS exhibit was never executed. A term TERMINATED IN ITS ENTIRETY by the amendment is not a term of this Agreement -- and its whole Article is still printed above. A term ADMINISTERED UNDER A SEPARATE AGREEMENT is somebody else's paperwork. All three carry codes in the same format as a real term, and putting one on the worksheet puts a rate in front of a reviewer that this Agreement does not carry.
2. THE AMENDMENT IS OPERATIVE AND THE SCHEDULE IS NOT. Where an executed amendment restates a term's rate basis, THE AMENDMENT'S BASIS IS `contract_rate_basis` and the Article's original basis is history. The SCHEDULE OF TERMS AT EXECUTION was cut when the Agreement was signed and is never re-cut, so on every amended term it shows the OLD basis. Do not read the answer off it. Read the amendment first, then the Article.
3. REPORT WHAT EACH DOCUMENT STATES. The `contract_` fields are readings of the Agreement and the `config_` fields are readings of the configuration extract. Never let one fill in for the other. If an Article exists and states no rate basis at all -- referring instead to a manual, to published payment policies, or to rates in effect between the parties -- then `contract_rate_basis` is 'silent', and it stays 'silent' however confidently the configuration carries a basis.
4. `rate_check` IS DECIDED ONLY BY THE RULEBOOK BELOW. Work through the steps IN ORDER and STOP at the first that fires:
   a. contract_rate_basis 'no_article' -> 'no_contract_term'.
   b. contract_rate_basis 'silent'     -> 'not_determined'. THIS FIRES WHATEVER THE CONFIGURATION CARRIES.
   c. config_rate_basis 'absent'       -> 'not_loaded'.
   d. the two bases differ             -> 'basis_differs'.
   e. contract_carve_out 'stated' and config_carve_out 'absent' -> 'carve_out_missing'.
   f. otherwise                        -> 'agrees'.
5. THE HEADLINE AGREEING IS NOT THE TERM AGREEING. An Article that sets a rate basis and then, several paragraphs further down, EXCLUDES a site, a sub-category or a circumstance from that basis has stated a carve-out. If the configuration row carries no carve-out, the term is 'carve_out_missing' even though the rate bases are identical. An annual ESCALATOR is not a carve-out and neither is a claims-submission, credentialling, notice or operating-review clause.
6. `date_check` IS DECIDED ONLY FROM THE TWO EFFECTIVE BASES: contract 'silent' -> 'not_determined'; config 'absent' -> 'not_loaded'; different -> 'basis_differs'; same -> 'agrees'. The basis is the EVENT the Agreement ties the term to -- services rendered, or claims received -- and not the date itself. A term the Agreement ties to the date of service and the configuration loads as claim-received is 'basis_differs'.
7. 'not_determined' IS A REAL ANSWER AND YOU ARE EXPECTED TO USE IT. A worksheet that never reaches for it is guessing, and a call recorded as settled is a call nobody re-reads.
8. Copy `term_code`, `service_category` and `contract_rate_verbatim` verbatim. Use the exact allowed value for every field that lists them, and return every field for every term.

CONTRACT-LOAD RULEBOOK (the authority for `rate_check` and `date_check`; this is an ILLUSTRATIVE rulebook written for this kit, and it reproduces no statute, no regulation and no payer's configuration manual)

WHAT COUNTS AS A TERM ON THE WORKSHEET
  - A TERM belongs on the worksheet if the Agreement's OPERATIVE Articles state it, or the configuration extract carries a loaded row for it. Both sides put a term on the worksheet; neither side alone keeps one off.
  - A term in the PROPOSED TERMS exhibit is not executed. Its article is written out in full and its code is in the same format as an executed term, and it is not on the worksheet.
  - A term TERMINATED IN ITS ENTIRETY by an amendment is not on the worksheet. Its Article is still printed above the amendment; only the schedule's status column and the amendment say it is gone.
  - A term ADMINISTERED UNDER A SEPARATE AGREEMENT that this Agreement only refers to is not on the worksheet. It is somebody else's paperwork.
  - A configuration row whose code appears in NO operative Article is still on the worksheet. That is the phantom, and it is the row a reviewer most needs to see: the claims system is configured to pay something this Agreement does not say.

THE SIX STATED FACTS, three read off the AGREEMENT and three off the CONFIGURATION
  contract_rate_basis:
    percent_of_schedule  The operative Article states the rate as a percentage of a fee schedule.
    case_rate            The operative Article states a single fixed amount for a completed episode.
    per_diem             The operative Article states a fixed amount for each day of an admission.
    silent               The Article for this term exists and states NO rate basis at all. A reference to a manual, a policy or 'rates in effect between the parties' names no basis and is silent.
    no_article           This Agreement has no operative Article for this term at all. Only the configuration carries it.
  config_rate_basis:
    percent_of_schedule  The configuration extract's row for this term carries a percentage-of-schedule basis.
    case_rate            The configuration extract's row for this term carries a case-rate basis.
    per_diem             The configuration extract's row for this term carries a per-diem basis.
    absent               The configuration extract carries no row for this term, or its rate-basis cell is blank.
  contract_carve_out:
    stated               The Article states an exclusion, exception or carve-out from the rate basis it just set — a site, a sub-category or a circumstance that is NOT payable on that basis.
    silent               The Article states no exclusion, exception or carve-out.
  config_carve_out:
    loaded               The configuration extract's row for this term carries a carve-out.
    absent               The configuration extract's row for this term carries no carve-out, or there is no row.
  contract_effective_basis:
    service_date         The operative Article ties the effective date to the DATE OF SERVICE — services rendered on or after a date.
    receipt_date         The operative Article ties the effective date to the DATE THE CLAIM IS RECEIVED — claims received on or after a date.
    silent               The Article states no effective basis at all, or there is no Article.
  config_effective_basis:
    service_date         The configuration extract's row carries a date-of-service effective basis.
    receipt_date         The configuration extract's row carries a claim-received effective basis.
    absent               The configuration extract carries no row for this term, or its effective-basis cell is blank.

WHICH ARTICLE IS OPERATIVE
  AN AMENDMENT IS PART OF THE AGREEMENT AND THE SCHEDULE IS NOT. Where an executed amendment restates a term's rate basis, the amendment's basis is the operative one and the Article's original basis is history. The SCHEDULE OF TERMS at the head of the extract was prepared at execution and is not re-cut when an amendment lands, so it shows the ORIGINAL basis. Reading the schedule and the configuration side by side is the fastest way to do this work and it is wrong on exactly the terms that matter.

rate_check -- work through IN ORDER, stop at the first that fires
  1. If contract_rate_basis is `no_article`, the call is `no_contract_term`. The configuration carries a term this Agreement does not state. Stop here.
  2. Otherwise, if contract_rate_basis is `silent`, the call is `not_determined`. The Agreement has an Article for this term and that Article never states a rate basis, so the paperwork does not settle it — WHATEVER the configuration carries. Stop here.
  3. Otherwise, if config_rate_basis is `absent`, the call is `not_loaded`. The Agreement states a basis and the configuration carries none.
  4. Otherwise, if config_rate_basis is not the same value as contract_rate_basis, the call is `basis_differs`.
  5. Otherwise, if contract_carve_out is `stated` and config_carve_out is `absent`, the call is `carve_out_missing`. The headline basis agrees and the exception does not, which is the whole reason this check is not a two-column diff.
  6. Otherwise the call is `agrees`.

date_check -- work through IN ORDER, stop at the first that fires
  1. If contract_effective_basis is `silent`, the call is `not_determined`. Stop here — a configuration that carries an effective basis the Agreement never states is not evidence about the Agreement.
  2. Otherwise, if config_effective_basis is `absent`, the call is `not_loaded`.
  3. Otherwise, if config_effective_basis is not the same value as contract_effective_basis, the call is `basis_differs`. A service-date term loaded as a claim-received term is the commonest of these and it is not a rounding difference: the two select different work.
  4. Otherwise the call is `agrees`.

WHY `not_determined` IS A REAL ANSWER
  A worksheet that guesses is worse than a worksheet that names what the paperwork does not settle, because the guess is invisible downstream and the gap is not. `not_determined` is a finding ABOUT THE AGREEMENT and is the correct, complete answer whenever the Agreement does not say. It is never a way of saying the reader was unsure, and it is never overridden by what the configuration happens to carry — a loaded value is a claim about the claims system, not about the contract.

Return these:
- agreement_ref (string) -- the agreement reference from the header, verbatim (for example PC-2019-441)
- terms (array of objects) -- one object per WORKSHEET term, each carrying:
    - term_code (string) -- the term code, verbatim (for example TRM-4821). THIS IS THE KEY -- one object per term the OPERATIVE Agreement states, plus one for every configuration row whose code appears in no operative Article. A term in the PROPOSED TERMS exhibit, a term terminated in its entirety by an amendment, and a term administered under a separate agreement are NOT on the worksheet and must not appear
    - service_category (string) -- the service category this term covers, copied verbatim from the Article heading or, for a configuration-only term, from the configuration extract row
    - contract_rate_basis (enum) one of: percent_of_schedule, case_rate, per_diem, silent, no_article -- the rate basis the OPERATIVE Agreement states for this term. ⚠︎ OPERATIVE MEANS AFTER ANY EXECUTED AMENDMENT. Where an amendment restates the rate basis, the amendment's basis is the answer and the Article's original basis is history. The SCHEDULE OF TERMS was cut at execution and is never re-cut, so it shows the ORIGINAL basis -- do not read the answer off it. 'silent' when the Article exists and names no basis at all: a reference to a manual, a policy or 'rates in effect between the parties' names no basis. 'no_article' when this Agreement has no operative Article for the term and only the configuration carries it
    - config_rate_basis (enum) one of: percent_of_schedule, case_rate, per_diem, absent -- the rate basis the CONFIGURATION EXTRACT carries for this term. 'absent' when the extract has no row for the term, or the row's rate-basis cell is blank. Report what is loaded; this is the thing under review and is never evidence about the contract
    - contract_carve_out (enum) one of: stated, silent -- does the Article state an exclusion, exception or carve-out FROM the rate basis it just set -- a site, a sub-category or a circumstance that is not payable on that basis? ⚠︎ THE CARVE-OUT IS TYPICALLY SEVERAL PARAGRAPHS BELOW THE RATE SENTENCE and is the sentence most often skipped. An annual escalator is NOT a carve-out. 'silent' where the Article states no exclusion, or where there is no Article
    - config_carve_out (enum) one of: loaded, absent -- does the configuration extract's row for this term carry a carve-out? 'absent' when the carve-out cell reads none, is blank, or there is no row
    - contract_effective_basis (enum) one of: service_date, receipt_date, silent -- what event the operative Agreement ties this term's effective date to. 'service_date' for services rendered or provided on or after a date. 'receipt_date' for claims received or submitted on or after a date. 'silent' where the Article states no effective basis, or where there is no Article. A stated calendar date with no event named is 'silent' -- the basis is the EVENT, not the date
    - config_effective_basis (enum) one of: service_date, receipt_date, absent -- what event the configuration extract's row ties the term to. 'absent' when the extract has no row for the term, or the row's effective-basis cell is blank
    - superseded_by_amendment (enum) one of: yes, no -- does an executed amendment in this extract restate this term's rate basis? Recorded because it is the single most expensive thing to miss on this work -- it is reported here so the worksheet shows the amendment was READ, whether or not the configuration caught up with it
    - contract_rate_verbatim (string) -- the OPERATIVE rate sentence, copied verbatim from the Article or, where an amendment restated the basis, from the amendment. null where the Agreement states no rate basis for this term at all
    - rate_check (enum) one of: agrees, basis_differs, carve_out_missing, not_loaded, not_determined, no_contract_term -- THE RULEBOOK'S CALL, not your own view. Work through the rate_rule steps IN ORDER and stop at the first that fires. Do not reason to a call and then look for the step that agrees with it
    - date_check (enum) one of: agrees, basis_differs, not_loaded, not_determined -- THE RULEBOOK'S CALL for the effective basis, asked of every term independently of its rate call

Return a JSON object with exactly these top-level keys: agreement_ref, terms
`terms` is an array. Return it empty only if the extract carries no worksheet term at all.

CONTRACT AND CONFIGURATION EXTRACT
----------------------------------
RILLWATER HEALTH PLAN
PROVIDER CONTRACTING — CONTRACT AND CONFIGURATION REVIEW EXTRACT

SYNTHETIC RECORD. This is generated test paperwork. No member, patient or claim record of
any kind appears in it; the payer, the provider group, every code and every amount are
invented. It reproduces no statute, regulation, standard or real rate schedule.

Provider group:      Pinehollow Regional Group
Agreement ref:       PC-2022-737
Extract id:          CLQ-0021
Configuration load:  CFG-2022-4659

This extract pairs the executed Agreement with the configuration record loaded from it for
review. The Schedule below is the execution snapshot; the Articles set out each term; any
Amendment follows them; the Configuration Extract at the foot is what the claims system
carries today.

SCHEDULE OF TERMS AT EXECUTION
--------------------------------------------------------------------------------------------
CODE        SERVICE CATEGORY                RATE BASIS AT EXECUTION   STATUS
--------------------------------------------------------------------------------------------
TRM-2660    Sub-acute inpatient             percent of schedule       executed
TRM-3074    Outpatient surgical             not stated                executed
TRM-2034    Infusion services               percent of schedule       executed
TRM-9832    Skilled nursing                 per diem                  executed
TRM-1547    Outpatient therapy              case rate                 executed
--------------------------------------------------------------------------------------------

ARTICLE 1 — Sub-acute inpatient (TRM-2660)

  Compensation for this category is 97 per cent of the Payer's then-current fee schedule for the
  category.

  Provider shall maintain for the term of this Agreement the credentialling records this
  Agreement requires for the category.

  Provider shall submit claims in the format agreed between the parties and within the
  timely-filing window stated in Article 9.

ARTICLE 2 — Outpatient surgical (TRM-3074)

  Rates for this category are those in effect between the parties and are administered under the
  Payment Policy Manual referenced in Article 1.

  The parties shall meet annually to review the utilisation reporting produced for this category
  under Article 11.

  Either party may request a joint operating review of this category on thirty days written
  notice to the other.

  The terms of this Article take effect for claims received by the Payer on or after 15 July
  2020.

ARTICLE 3 — Infusion services (TRM-2034)

  Covered services in this category are payable at 119 per cent of the Payer's fee schedule then
  in effect.

  Provider shall maintain for the term of this Agreement the credentialling records this
  Agreement requires for the category.

  Provider shall notify the Payer of any change to the sites at which this category is
  delivered, within thirty days of the change.

  The terms of this Article take effect for claims received by the Payer on or after 1 August
  2025.

ARTICLE 4 — Skilled nursing (TRM-9832)

  Covered services in this category are payable at a per diem of USD 3,400 for each day of an
  admission.

  Provider shall submit claims in the format agreed between the parties and within the
  timely-filing window stated in Article 9.

  Provider shall notify the Payer of any change to the sites at which this category is
  delivered, within thirty days of the change.

  Either party may request a joint operating review of this category on thirty days written
  notice to the other.

  Implant devices supplied in this category are excluded from the rate basis stated above and
  are payable separately under Article 12.

  The terms of this Article take effect for services delivered on or after 1 October 2025.

ARTICLE 5 — Outpatient therapy (TRM-1547)

  Covered services in this category are payable at a case rate of USD 46,500 for each completed
  episode.

  The rate basis stated above escalates by 1.9 per cent on each anniversary of the Effective
  Date.

  Provider shall submit claims in the format agreed between the parties and within the
  timely-filing window stated in Article 9.

  Either party may request a joint operating review of this category on thirty days written
  notice to the other.

  Provider shall maintain for the term of this Agreement the credentialling records this
  Agreement requires for the category.

  Services in this category delivered at the Provider's Weldon Street site are excluded from the
  rate basis stated above and are payable under Article 12.

  The terms of this Article take effect for services delivered on or after 15 March 2022.

TERMS ADMINISTERED UNDER A SEPARATE AGREEMENT

  Ambulance transport (TRM-9347) is administered under the separate agreement between the
  parties dated 2011 and is neither established nor varied by this Agreement. Its rate basis is
  recorded in that agreement.

AMENDMENT — EXECUTED SUBSEQUENT TO THE ARTICLES ABOVE

  The rate basis for Skilled nursing (TRM-9832) is restated as follows, in place of the basis
  stated in the Article above. Compensation for this category is 96 per cent of the Payer's
  then-current fee schedule for the category.

CONFIGURATION EXTRACT — CLAIMS SYSTEM (LOAD CFG-2022-4659)

  This is what the claims system carries for this Agreement today. It is a configuration
  record, not a claim and not a payment: no member, patient or claim appears in it.

--------------------------------------------------------------------------------------------------------
CODE        SERVICE CATEGORY                RATE BASIS              CARVE-OUT           EFFECTIVE BASIS
--------------------------------------------------------------------------------------------------------
TRM-2660    Sub-acute inpatient             percent of schedule     none                claim received
TRM-3074    Outpatient surgical             per diem                none                (not loaded)
TRM-9832    Skilled nursing                 per diem                none                date of service
TRM-1547    Outpatient therapy              case rate               none                date of service
--------------------------------------------------------------------------------------------------------

Raw responseThe raw response

{
  "agreement_ref": "PC-2022-737",
  "terms": [
    {
      "term_code": "TRM-2660",
      "service_category": "Sub-acute inpatient",
      "contract_rate_basis": "percent_of_schedule",
      "config_rate_basis": "percent_of_schedule",
      "contract_carve_out": "silent",
      "config_carve_out": "absent",
      "contract_effective_basis": "silent",
      "config_effective_basis": "receipt_date",
      "superseded_by_amendment": "no",
      "contract_rate_verbatim": "Compensation for this category is 97 per cent of the Payer's then-current fee schedule for the category.",
      "rate_check": "agrees",
      "date_check": "not_determined"
    },
    {
      "term_code": "TRM-3074",
      "service_category": "Outpatient surgical",
      "contract_rate_basis": "silent",
      "config_rate_basis": "per_diem",
      "contract_carve_out": "silent",
      "config_carve_out": "absent",
      "contract_effective_basis": "receipt_date",
      "config_effective_basis": "absent",
      "superseded_by_amendment": "no",
      "contract_rate_verbatim": null,
      "rate_check": "not_determined",
      "date_check": "not_loaded"
    },
    {
      "term_code": "TRM-2034",
      "service_category": "Infusion services",
      "contract_rate_basis": "percent_of_schedule",
      "config_rate_basis": "absent",
      "contract_carve_out": "silent",
      "config_carve_out": "absent",
      "contract_effective_basis": "receipt_date",
      "config_effective_basis": "absent",
      "superseded_by_amendment": "no",
      "contract_rate_verbatim": "Covered services in this category are payable at 119 per cent of the Payer's fee schedule then in effect.",
      "rate_check": "not_loaded",
      "date_check": "not_loaded"
    },
    {
      "term_code": "TRM-9832",
      "service_category": "Skilled nursing",
      "contract_rate_basis": "percent_of_schedule",
      "config_rate_basis": "per_diem",
      "contract_carve_out": "stated",
      "config_carve_out": "absent",
      "contract_effective_basis": "service_date",
      "config_effective_basis": "service_date",
      "superseded_by_amendment": "yes",
      "contract_rate_verbatim": "Compensation for this category is 96 per cent of the Payer's then-current fee schedule for the category.",
      "rate_check": "basis_differs",
      "date_check": "agrees"
    },
    {
      "term_code": "TRM-1547",
      "service_category": "Outpatient therapy",
      "contract_rate_basis": "case_rate",
      "config_rate_basis": "case_rate",
      "contract_carve_out": "stated",
      "config_carve_out": "absent",
      "contract_effective_basis": "service_date",
      "config_effective_basis": "service_date",
      "superseded_by_amendment": "no",
      "contract_rate_verbatim": "Covered services in this category are payable at a case rate of USD 46,500 for each completed episode.",
      "rate_check": "carve_out_missing",
      "date_check": "agrees"
    }
  ]
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch a provider contract loaded at the wrong rate — 50 contract extracts. One model answered, and every answer was then graded Five different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran these runs publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersFive ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The two-table floor (evals/baseline.py) and the scorer (evals/judge.py) are both pure code and cost $0.00 to run against any result set. The figure above is the ONE SHIPPED run's own token counts (50 extracts) priced at Google Gemini 3 Flash's published rate — the same basis cost_per_query_usd uses. ⚠︎ IT IS NOT WHAT BUILDING THIS KIT COST, AND THE GAP IS BIGGER HERE THAN ON ITS SIBLINGS. 166 live calls were fired in total: a 6-extract calibration ($0.1113), THREE full 50-extract runs rather than one — the first contaminated by a selection leak ($1.1783), the second by a 24,000-token ceiling that lost a document ($1.1068), and the third the one published ($1.0308) — four prompt-split calls at max_tokens=1 ($0.0058), two worked-example captures ($0.0442), one ceiling probe ($0.0296) and three live UI screenshots whose tokens were not recorded (about $0.068 at the worked example's measured size). All of it projects to roughly $3.57 on this card.

Cost driversWhat actually moves the bill

• The fixed prefix — system prompt (1,025 tokens), contract-load rulebook (1,394) and field schema (1,022) — is 3,441 of 4,783 tokens on the worked example, 71.9 pct, and is byte-identical on every one of the 50 calls. The two documents themselves are 1,342 tokens. The bill is the rulebook and the instructions, not the paperwork.
• Provider-side reasoning. 262,528 of the run's 303,433 output tokens (86.5 pct) were reasoning rather than the JSON worksheet, which is about 800 tokens of actual content per extract. Reasoning was left at the provider's default and nothing here has measured what turning it off would cost in accuracy — and on a kit whose hard half is holding an amendment, an Article and a table in mind at once, that is a more interesting question than it is on a flat extraction kit.
• Output length: the model returns a full ten-field object for every term, including the operative rate sentence copied back verbatim, whatever the extract says.

Your volumeWhat it costs at your volume

Linear in extracts: each call is independent, carries the same fixed prefix and shares nothing with its neighbours, so 500 extracts cost ten times 50. Wall clock is not linear in the same way — the run took 220.7 seconds at 12 concurrent workers rather than the ~39 minutes it would take serially, and EVAL_WORKERS is the knob. Nothing amortises: there is no index and no cache. ⚠︎ AND THE RULEBOOK DOES NOT SCALE LINEARLY, IT SCALES WORSE: it is sent whole in every call, so a real payer's configuration manual — dozens of rate bases, each with its own definition and its own exception logic — multiplies the input side of every call at once. At that size the rulebook has to be split, only the rules relevant to the bases actually present sent per call, and this kit's prompt would have to be rebuilt around it.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 contract-and-configuration extracts, one completion call each, on one tier. There is no second tier on this kit and therefore no second row.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the extract into addressable sections, pure code — one per written-up term, plus the schedule, the amendment, the decoy blocks and the CONFIGURATION EXTRACT that is the other half of the comparison

# Cut a contract-and-configuration extract into addressable sections. Pure code — no model.

FIXED_HEADINGS = (
CONFIGURATION = "CONFIGURATION EXTRACT — CLAIMS SYSTEM"
def _marks(text):
def sections(text):
def term_codes(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections are sent, pure code — matched by PREFIX because every extract's Article sections are named after its own term codes. The three decoy-bearing sections and both sides of the comparison are deliberately SENT; the CONTRACTING CONTACT AND NEGOTIATION RECORD is mapped by nothing and never leaves the machine, and the whole-document fallback is barred from reaching it

You change it to: map fields to your own layout's headings, including the per-term prefix; unmatched falls back to the whole document MINUS the NEVER_SENT list. NEVER_SENT is the set of sections no field asks for and which therefore never leave the machine

# Pick which sections of an extract are sent. Pure code -- the last deterministic step before the

HEADER = "header"
SCHEDULE = "SCHEDULE OF TERMS AT EXECUTION"
DRAFT = "EXHIBIT B — PROPOSED TERMS, NOT EXECUTED"
SEPARATE = "TERMS ADMINISTERED UNDER A SEPARATE AGREEMENT"
AMENDMENT = "AMENDMENT — EXECUTED SUBSEQUENT TO THE ARTICLES ABOVE"
CONFIG = CONFIGURATION
ARTICLE_PREFIX = "Article "
NEVER_SENT = ("CONTRACTING CONTACT AND NEGOTIATION RECORD",)
SECTION_HINTS = {

src/rulebook.pyrulebook

the two worksheet calls, loaded from data/rulebook.json — a six-step rate rule with a stopping order and a four-step date rule. Data, not code, so a forker can open it and disagree with it

# THE CONTRACT-LOAD RULEBOOK, loaded from data/rulebook.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
CONTRACT_RATE_BASES = ("percent_of_schedule", "case_rate", "per_diem", "silent", "no_article")
CONFIG_RATE_BASES = ("percent_of_schedule", "case_rate", "per_diem", "absent")
CONTRACT_CARVE_OUTS = ("stated", "silent")
CONFIG_CARVE_OUTS = ("loaded", "absent")
CONTRACT_EFFECTIVE_BASES = ("service_date", "receipt_date", "silent")
CONFIG_EFFECTIVE_BASES = ("service_date", "receipt_date", "absent")
RATE_CALLS = ("agrees", "basis_differs", "carve_out_missing", "not_loaded", "not_determined",

src/prompt.pyprompt

assemble one call per extract for every worksheet term, with the rulebook RENDERED into it from data/rulebook.json rather than restated in prose — a rule the model has never been shown is not a rule

# Assemble the worksheet prompt. ONE prompt per extract, every worksheet term in the reply.

SYSTEM = (
RULEBOOK_TEXT = rulebook_block()
def field_schema(packet_fields, term_fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key — plus the pure-code business-condition check downstream: an extract carrying a term whose rate basis an executed amendment restated and whose configuration still carries a different basis goes back to contracting

You change it to: the review rule itself — this kit ships one condition (a superseded rate the configuration still carries) and a real review weighs volume, whether the amendment's effective date has passed and whether anything has actually been paid on the stale basis. It is deliberately NOT the same function as the rulebook lookup, so changing WHAT GETS REVIEWED does not change WHAT THE RULEBOOK SAYS

# Extract one contract-and-configuration worksheet: segment, select, prompt, one model call, then

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 48000
def load_fields():
def term_fields(fields=None):
def packet_fields(fields=None):
def load_doc(doc_id):
def documents():

evals/judge.pyjudge

score identification, the extracted cells, the two calls and the determinacy split, all separately, pure code — plus a no-gold consistency diagnostic

# Score a contract-load worksheet run. PURE CODE -- gold is exact and every answer is one value,

def norm(v):
def code_norm(v):
def equal(got, want):
def _rows_of(rec):
def score_identification(records, golds):
CELL_FIELDS = ("service_category", "contract_rate_basis", "config_rate_basis",
def score(fields, records, golds):
CALLS = {"rate_check": RB.RATE_CALLS, "date_check": RB.DATE_CALLS}
CONFIDENT = tuple(v for v in set(RB.RATE_CALLS) | set(RB.DATE_CALLS) if v != "not_determined")

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 4820 input and 6068 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's extracts are entirely synthetic (tools/build_corpus.py, seed 20260822). NO MEMBER DATA AND NO PATIENT DATA APPEARS ANYWHERE IN THIS KIT. Every document is contract and configuration paperwork generated from a fixed seed: an executed payer-provider agreement and the configuration record loaded from it. There is no member, no patient, no claim for a real person, no diagnosis, no clinical detail and no identifier belonging to anyone receiving care, and every payer, provider group, term code and amount is invented. evals/check_labels.py asserts it over every line of every file before any run may spend. No real payer, provider group, health system, clinic, person or product is named anywhere in the kit, and no statute, regulation, standard or code scheme is reproduced or named. The only outbound traffic the kit makes is one chat-completion request per extract to the configured provider, carrying the mapped sections of one extract plus the fixed prompt and the shipped rulebook — and the CONTRACTING CONTACT AND NEGOTIATION RECORD, which is the one part of a real pairing a contracting office would most mind sending anywhere, is mapped by no field and never leaves the machine. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser.

The experimentWe did not attack it — and one of the three boundaries that hold did not, until a gate was widened

Two boundaries were confirmed by reading the code: no code path determines payment, adjudicates, reprocesses or writes anything, and the rulebook lookup and the review rule read only enums, so no clause text can reach either. Two more are confirmed by CHECKS RATHER THAN BY READING — evals/check_labels.py asserts, before any run may spend, that no member or patient vocabulary appears on any line of any file, and that the negotiation record is never sent while every decoy section and both sides of the comparison are. The second of those did not hold when it was first written: it sampled one file, passed, and the record was reaching the provider on 17 extracts. The fourth boundary is open. An indirect prompt injection needs a section an outside party authored, and an agreement is exactly that. Confirmed by reading the code and by two pre-flight assertions, not by an attack run, on 2026-08-22 — no attack run was fired; see redteam.why.

The first three boundaries hold, and the second and third are confirmed by assertions that run before every scored run rather than by reading the code alone. The third is published with its own failure attached, because it did not hold until a gate was widened. The fourth is open and is named as open — an agreement is a document the other side helped write, which makes an instruction hidden in a clause exactly the attack this kit's shape invites, and nothing here has tried it.

The result0 member or patient records in the corpus by construction and by assertion, 0 attack trials, and three of four boundaries checked here hold on evidence — two of them by checks that run before every scored run rather than by reading the code. One of those three did NOT hold until the gate was widened from one file to fifty.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted decoys measure whether a model reads the AMENDMENT to decide what is operative when the schedule and the configuration point the other way, and whether it declines a call the Agreement does not settle. Neither measures whether it obeys an instruction hidden in the same document. Those are different failures and only the first two are measured here.

HonestyWhat this does not prove

• Whether an indirect prompt injection in an Article or an amendment could move a call. No attack run exists.
• Whether the kit behaves safely against a hostile provider — a response body crafted to break the list parsing in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed document, and nothing beyond that was tested.
• Whether a real pairing would carry anything sensitive this kit mishandles beyond the negotiation record. The corpus has no personal data by construction; a real agreement carries named signatories and negotiated rates both sides consider confidential, and none of that path is exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_contracting_review fires when the extract carries at least one term whose rate basis an EXECUTED AMENDMENT restated — superseded_by_amendment == "yes" — and whose rate call, RE-DERIVED by the shipped rulebook from that term's own extracted bases rather than taken from the model's stated call, is "basis_differs". Both inputs come out of the same reply; the rule is run afterwards in pure code, over whatever the model returned, and never over gold. A reply carrying no term the rule can read returns None rather than False — an unknown is not a pass.

src/extract.py::compute(), called by extract() on every extract, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by identical code. The rulebook lookup it reads lives in a SEPARATE module, src/rulebook.py, which is also what the corpus generator wrote gold with and what src/prompt.py states to the model — one definition, three readers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a payment determination, a claims adjudication, a reprocessing instruction or a configuration change. Nothing here determines payment, adjudicates a claim, authorises a reprocessing, changes a configuration or notifies anybody. rate_check and date_check are proposals with the rulebook's own reasoning attached, and contracting decides.
• It is NOT a check on whether the extracted facts are right. If the model misreads the amendment and then applies the rulebook correctly to the misreading, the extract is flagged — or not flagged — on a wrong fact, and this rule cannot tell. THAT IS NOT HYPOTHETICAL HERE: the one scoring error in this kit's history is exactly that shape, a configuration row the model invented and then reasoned about faultlessly, and the consistency diagnostic in evals/judge.py caught 0 of it.
• It is NOT any payer's configuration policy. A superseded rate the configuration still carries is this kit's own simplification, invented for this corpus. No statute, regulation, standard, payer manual or contracting policy was consulted, and none is reproduced. A real review weighs how much volume runs through the category, whether the amendment's effective date has passed, and whether anything has actually been paid on the stale basis.
• It is NOT a judgement about the rulebook. The flag reads the calls the rulebook produced; if the rulebook is wrong, the flag is confidently wrong with it. The shipped rulebook is illustrative and nobody qualified has reviewed it.
• It is NOT about AMOUNTS. A term configured to the right basis at the wrong percentage, case rate or per diem never fires this flag and is wrong in the claims system.

WatchedWhat is watched, and why that one

The kit measures 14 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 7 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 50, extraction_cells 2800, failures 0, refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

extraction · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on documents, extraction_cells, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

extraction · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 7 chips that all say so.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/extract.py, on any change to data/rulebook.json (which moves the calls the flag reads), on any change to data/fields.json's allowed values for the rate bases or the amendment flag, and on any corpus regeneration. Changing WHAT GOES BACK TO CONTRACTING is a policy change and must be re-scored, even though it never changes what the rulebook says.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the rulebook is a JSON file read with json, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per extract — segment, select, prompt, one call, parse, compute — with no branch, no loop and no agent. The only structural difference from its flat siblings is that the REPLY is a list, which changes the scorer and not the pipeline.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain — the JSON extraction from a fenced reply, the list parsing, the backoff policy, the section regexes, the prefix matcher and the .env reader are all code somebody has to own. This kit paid two instalments of that during its own build, and both are recorded rather than tidied away: the selection fallback that defeated NEVER_SENT, and a span locator that could not find a hard-wrapped sentence a model had copied verbatim (fixed by collapsing whitespace in src/segment.py::locate).
• No caching layer, so the 3,441-token fixed prefix is paid on every one of the 50 calls.
• No queue, no persistence and no retry beyond the adapter's four backoff attempts. A run that fails halfway leaves nothing behind but the printed lines.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-contract-load-qa on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b000-contract-load-qa-tablediff, t000-contract-load-qa-stub recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — blocking is computed in memory each run and is the only reduction step.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. NO MEMBER DATA AND NO PATIENT DATA APPEARS ANYWHERE IN THIS KIT. Every document is contract and configuration paperwork generated from a fixed seed: an executed payer-provider agreement and the configuration record loaded from it. There is no member, no patient, no claim for a real person, no diagnosis, no clinical detail and no identifier belonging to anyone receiving care, and every payer, provider group, term code and amount is invented. evals/check_labels.py asserts it over every line of every file before any run may spend. No real procedure code, rate schedule or code scheme appears either, real or invented to look real. The contract-load rulebook this kit ships (data/rulebook.json) is ILLUSTRATIVE and is not an authority. It was written for this kit. It reproduces no statute, no regulation, no standard, no payer's configuration manual and no provider group's contracting policy, and it names none of them. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineWhich terms reached the worksheet

Did the run list exactly the terms this worksheet is owed? The worksheet set is the UNION of the Agreement's operative Articles and the configuration extract's rows, so both sides can put a term on it and neither alone keeps one off. Micro precision and recall over term codes, with the two directions named and never averaged: a MISSED term is a real one left off, and a PHANTOM term is one on the worksheet that belongs to neither document. ⚠︎ A CONFIGURATION ROW WITH NO ARTICLE IS NOT A PHANTOM — it is the finding, and leaving it off is a miss rather than a saving, which is why it is counted separately as missed_phantom_row.

MethodHow the test was run

evals/judge.py::score_identification, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, which records the worksheet terms AND the decoy codes for every extract; evals/check_labels.py asserts before any run may spend that every decoy code is in the file, collides with no worksheet term, and that each of the three kinds clears a floor of 8.

How tight can the band be? No tolerance band — a code is on the worksheet or it is not, compared after whitespace stripping and upper-casing.

Cadence: Re-score on any change to data/rulebook.json's what_is_a_term_here, to src/prompt.py's rule 1, to the corpus's decoy composition, or to MAX_TOKENS.

The decisionWhen to reach for it

In one linePer-(term, field) exact match, light normalisation

Does the model's value for each of the ten per-term fields match gold, after trimming whitespace and punctuation, plus one agreement reference per extract? THE DENOMINATOR IS GOLD'S TERMS, NOT THE RUN'S: a gold term the run never returned scores as a miss on all ten of its cells rather than disappearing, so a run cannot raise this number by returning less. That is not hypothetical here — it is exactly what held the second run to 97.46 pct when one document was lost to the output ceiling.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every run, and the same logic the free floor is scored by — a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same values the two documents state; evals/check_labels.py asserts every field is populated and in its allowed set on every one of the 275 terms, that contract_rate_verbatim is null exactly where the Agreement states no basis, and that every value is stated in the file it labels, before any run is allowed to spend.

How tight can the band be? There is no tolerance band — it is exact match after trimming whitespace and punctuation, never a continuous score to round.

Cadence: Re-score on any change to data/rulebook.json, src/prompt.py, data/fields.json or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe two rulebook calls, six-way and four-way

Did the run land on the same calls the shipped rulebook produces — agrees / basis_differs / carve_out_missing / not_loaded / not_determined / no_contract_term for the rate, and agrees / basis_differs / not_loaded / not_determined for the effective basis? SCORED ON MATCHED TERMS ONLY, and saying so is half of what makes the number honest: a call about a term the run never found has already been counted, once, as an identification failure, and counting it again here would let one mistake sink two graders.

MethodHow the test was run

evals/judge.py::score_calls, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's calls are re-derived inside the grader by the same lookup the kit publishes — src/rulebook.py, reading data/rulebook.json — so the truth this grades against can never be a separately-typed label that drifted from the rule.

How tight can the band be? No threshold — each call is one of a closed set, and a reply returning none of them is counted as unanswered rather than folded into a correct cell.

Cadence: Re-run whenever data/rulebook.json or src/rulebook.py changes, whenever the corpus is regenerated, and on any provider or model change.

The decisionWhen to reach for it

In one line⚑ The headline — how often it says the Agreement does not settle it

Of the calls the Agreement does NOT settle, how many did the run answer with a confident value anyway? That is overconfident. Its mirror, overcautious — declining a call the Agreement does settle — is counted and published beside it, because the two costs are different and do not net against each other. On this corpus the tempting wrong answer is always sitting right there in the configuration table, which is what makes the number worth measuring rather than asserting.

MethodHow the test was run

evals/judge.py::score_calls, the determinacy block, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: gold's not_determined calls, which are the rulebook's own output wherever the Agreement states no rate basis or no effective basis — never a judgement that the corpus was unclear.

How tight can the band be? No band — it is a count over the calls gold records as not_determined, and a reply that answered nothing is counted as unanswered rather than as caution.

Cadence: Re-run on any change to the prompt's rules 3, 6 or 7, to data/rulebook.json, or to the corpus's unstated-basis composition.

The decisionWhen to reach for it

In one lineneeds_contracting_review against the same rule run over gold

Did the pure-code business condition fire on exactly the extracts where it should? compute() is run over whatever the model returned and compared against the same function run over gold's own terms. A reply carrying no readable term returns None and is scored as unanswered rather than folded into a true negative — an unknown is not a pass.

MethodHow the test was run

evals/judge.py::score_flag, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute(), the SAME function, run over gold's own terms. It is not this page's reference standard — that role belongs to the per-(term, field) exact match — and it is a business condition rather than a reading, so it needs labels to score at all, unlike the consistency diagnostic beside it.

How tight can the band be? Binary. The 10–60 pct band is on the CORPUS's firing rate, not on the grader's score, and it is enforced in the gold gate rather than here.

Cadence: Re-score on any change to compute(), to data/rulebook.json, to the allowed values for the rate bases or the amendment flag, or on any corpus regeneration.

The decisionWhen to reach for it