The business caseThe problem this solves

Before a general contractor releases a progress payment, somebody has to decide whether the lien waivers on file actually cover the amounts about to be paid — for the subcontractor and for every lower tier beneath them. It is matching, plus arithmetic, plus type reasoning: the right KIND of waiver (conditional or unconditional, progress or final), for an amount that reaches, through a date that reaches, signed before rather than after the claim it would have to answer. Five ways to fail, in a priority order, per party, across four or five parties — and the payment coordinator's own note sits at the bottom of the page saying whether the file looked fine. Somebody opening each progress-payment package before the money moves, reading every party under it — the subcontractor and three or four lower tiers — and deciding, party by party, whether the waiver on file actually reaches the amount being released: is there a waiver at all, was it signed before the claim was even asserted, does a progress waiver's through-date reach the end of this period, does the amount reach, and is a conditional waiver still worth anything now that something has cleared against it. It is quick per party and it is every party on every package every cycle, and the part that goes wrong is not the arithmetic: it is checking the amount before checking the date, treating a final waiver's blank through-date as a gap, or reading the payment coordinator's own note first and letting it set the answer.

The inputThe actual payment packages

The corpus is 55 payment packages, 0.10 MB (txt 55). Plain text, one format, invented rather than fetched — a real lien-waiver payment package names a real project, real subcontractors and suppliers with real amounts owed to them, and a coordinator's own note about a live commercial dispute, and there is no public corpus of (payment package, correct coverage picture) pairs for the same reason there is no public corpus of bank statements. Generating it also makes the label mechanical: gold is the coverage rule run party by party over the values the package itself states, not somebody's reading of the note. And it is the only way to plant the four sharp cases — a notice arriving after signature, a final waiver's absent through-date, a period gap hiding behind a generous amount, and a joint check that clears on issue — in known quantities.

Package
-------
PA-2026-350

Project
-------
Stonebridge Logistics Campus

Prime Contractor
----------------
Northgate Builders

Subcontract Reference
---------------------
SC-F-6552, retainage 5 percent

Pay Application
---------------
Application 14

Period Through
--------------
2026-01-31

Payment Amount
--------------
524,467.17 USD

Prior Payment Cleared
---------------------
no

Release Status
--------------
scheduled

Waiver Coverage
---------------
Party 1: Marchand Roofing
  Tier: 1
  Amount due this application: 217,225.09 USD
  Preliminary notice on file: 2025-10-17
  Waiver on file: LW-88316, conditional progress waiver
  Waiver amount: 217,225.09 USD
  Waiver covers work through: 2026-01-31
  Waiver signed: 2026-02-04
  Joint check arrangement: no

Party 2: Ashby Insulation
  Tier: 2
  Amount due this application: 60,469.68 USD
  Preliminary notice on file: none
  Waiver on file: LW-84432, unconditional final waiver
  Waiver amount: 60,719.68 USD
  Waiver covers work through: n/a (final waiver, all work through completion)
  Waiver signed: 2026-02-11
  Joint check arrangement: no

Party 3: Calder Aggregates
  Tier: 2
  Amount due this application: 92,664.68 USD
  Preliminary notice on file: none
  Waiver on file: LW-44430, conditional progress waiver
  Waiver amount: 92,664.68 USD
  Waiver covers work through: 2026-03-02
  Waiver signed: 2026-02-02
  Joint check arrangement: no

Party 4: Belmont Conduit
  Tier: 3
  Amount due this application: 101,201.10 USD
  Preliminary notice on file: none
  Waiver on file: LW-78071, unconditional final waiver
  Waiver amount: 101,451.10 USD
  Waiver covers work through: n/a (final waiver, all work through completion)
  Waiver signed: 2026-02-05
  Joint check arrangement: yes

Party 5: Truxton Hoisting
  Tier: 2

The outcomeWhat a good result looks like

An eleven-field extracted record per package — including how many parties are uncovered, which is the first and why — plus one pure-code routing decision taken from two of those fields: a package with a gap that is still scheduled to go out is the one that has to be pulled back today. Nothing here releases a payment, withholds one, issues or requests a waiver, or determines anybody's lien rights.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Screening a draw cycle for pay applications whose waiver file is incomplete, before a payment coordinator opens them — the model, over the coordinator-note floor, by a wide margin
  100 pct coverage-gap accuracy on everything that answered, across three runs and both tiers, against the floor's 60.0 pct (22 of 55 wrong, 10 of them packages with a real gap called complete) — and, far more decisively, 93 of 93 gap attributions against the floor's 1 of 31.
• Deciding between the fast and the deliberating tier on cost, speed or accuracy — the fast tier
  They tie on every single published grader — 605 of 605 cells, 31 of 31 coverage verdicts, 31 of 31 attributions, 17 of 17 flags, on both. The deliberating tier costs about 15 pct more per package ($0.0047194 against $0.0040928), spends 20 pct more output tokens, and takes 2.1 times as long at the median (17,291 ms against 8,223). Nothing it bought showed up in a number.
• Deciding whether the reply's length is worth paying for — leave the reasoning on — but measure it before you decide, because this kit has not
  Replies run 468 to 1,812 output tokens for an eleven-key JSON record, three to four times what a sibling extraction kit needs for a record of the same shape, and output is 75 pct of the projected cost per package. The one wrong reply on record is also the only short one — 152 tokens, no working — which is suggestive and is a sample of one.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-waiver-match. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts, the 55-package picker populates and the field table draws its eleven empty rows; clicking Check coverage returns the no-key sentence rather than a stack trace, and the hold flag reads "not computed" rather than "no". python -m evals.check_labels passes with no network access at all, and python -m evals.run --baseline scores the free floor over all 55 packages offline.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per payment package

Current processWhat it replaces

Somebody opening each progress-payment package before the money moves, reading every party under it — the subcontractor and three or four lower tiers — and deciding, party by party, whether the waiver on file actually reaches the amount being released: is there a waiver at all, was it signed before the claim was even asserted, does a progress waiver's through-date reach the end of this period, does the amount reach, and is a conditional waiver still worth anything now that something has cleared against it. It is quick per party and it is every party on every package every cycle, and the part that goes wrong is not the arithmetic: it is checking the amount before checking the date, treating a final waiver's blank through-date as a gap, or reading the payment coordinator's own note first and letting it set the answer.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 0 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings; a package with none falls back to one whole-document segment. The three-to-five party blocks stay together inside the Waiver Coverage section, because the coverage question is asked across parties

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 55 packages cut into 605 sections in a few thousandths of a second, in process, with no model and no network.

LicenceLicence

MIT — this repository's own licence. Every project, contractor, supplier, waiver number, amount, date and coordinator note is invented; no real project, company, filed notice or waiver document is named or reproduced, and no jurisdiction's statutory waiver form is reproduced.

Bring your ownBring your own payment packages

Replace data/corpus/*.txt, write your own data/fields.json, and supply a gold record per package. SECTION_HINTS in src/select.py maps fields to section headings and will need editing for a different pay-application layout; when it does not match, selection falls back to the whole document — slower, more expensive, always correct. coverage_status() and compute() in src/extract.py are this kit's own invented coverage rule and routing policy and should be the first things you replace with your own counsel's requirements. Note the spannability declaration in data/fields.json: parties_uncovered is marked non-spannable because it is COUNTED rather than stated, and a type-derived rule would have located the digit 2 against the first party's "Tier: 2" line on every package.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 2,049 tokens assembled, 909 are instructions — 44% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on LWP-0005 — three calls at max_tokens=1, each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. kits/UC0045-waiver-match/results/tokens-p001-waiver-match.json.

Full promptThe prompt in full

You read a construction progress-payment package and report which parties on it are covered by a lien waiver and which are not. You return JSON and nothing else.

RULES, in order of importance:
1. If the package does not state a field, return null for it. Do not infer it and do not use what you know about the world.
2. `parties_uncovered`, `first_gap_party` and `first_gap_reason` are decided by applying ONE COVERAGE RULE to EACH party under `Waiver Coverage`, in document order, never by how the coordinator note reads. For each party, work through these five conditions IN ORDER and stop at the FIRST one that is true:
   a. `no_waiver_on_file` -- the party's `Waiver on file` line says none.
   b. `notice_after_waiver` -- the party has a `Preliminary notice on file` date AND it is LATER than the `Waiver signed` date. The waiver was signed before the claim was asserted, so it does not reach it. THIS OUTRANKS EVERYTHING BELOW -- it is true even when the waiver is unconditional, is for the full amount, and covers the whole period.
   c. `period_short` -- the party has a PROGRESS waiver (conditional progress or unconditional progress) whose `Waiver covers work through` date is EARLIER than the package's `Period Through` date. A FINAL waiver (conditional final or unconditional final) has NO through-date -- it covers all work through completion -- so a final waiver is NEVER `period_short`, and its "n/a" through-date is not a gap.
   d. `amount_short` -- the `Waiver amount` is LESS THAN that party's `Amount due this application`.
   e. `conditional_stale` -- the waiver is CONDITIONAL (conditional progress or conditional final) AND something has already cleared against it: either the package's `Prior Payment Cleared` is yes, OR that party's `Joint check arrangement` is yes. A joint check is negotiated by both payees, so it clears on issue -- a conditional waiver on a joint-check party is stale EVEN WHEN the package says the prior payment has not cleared.
   A party for which none of the five is true is COVERED.
3. THE ORDER IS PART OF THE RULE. More than one condition can be true of the same party, and only the FIRST one in the list above is reported. A waiver for twice the amount due whose through-date stops inside the period is `period_short`, not `amount_short`.
4. `parties_uncovered` is the COUNT of parties for which any condition fired. `first_gap_party` is the name of the FIRST such party in document order, and `first_gap_reason` is that party's own first-firing condition. When every party is covered, return 0, null and 'none' respectively.
5. THE COORDINATOR NOTE IS A FIELD TO COPY, NOT EVIDENCE ABOUT COVERAGE. A note that sounds settled does NOT mean every party is covered, and a note that sounds worried does NOT mean one is not. The party blocks, the period-through date and the prior-payment answer decide; the note is the coordinator's own remark and may disagree with them.
6. Copy values verbatim from the package wherever possible, and report payment_amount_usd as a bare number with no currency symbol, no thousands separators and no unit. Report first_gap_party as the party's name exactly as the package writes it, without the "Party N: " prefix.
7. Use the exact allowed value for a field that lists them.
8. Return every field named in the schema, even when the answer is null.

You are assembling evidence for a person, not authorising anything. Nothing you return releases a payment or decides anybody's lien rights.

Extract these fields:
- package_id (string) -- the payment package identifier, verbatim
- project_name (string) -- the project this pay application belongs to, verbatim
- pay_app_number (string) -- the pay application number, verbatim (e.g. 'Application 07')
- period_through (string) -- the through-date of the payment period, YYYY-MM-DD
- payment_amount_usd (number) -- the total being released on this pay application, as a bare number with no currency symbol, no thousands separators and no unit
- prior_payment_cleared (enum) one of: yes, no -- has the previous progress payment on this package already cleared?
- release_status (enum) one of: scheduled, on_hold -- is this pay application scheduled to be released this cycle, or already on hold?
- coordinator_note (string) -- the payment coordinator's own free-text note on the package, copied verbatim
- parties_uncovered (number) -- how many of the parties listed under Waiver Coverage are NOT covered by a waiver, applying the coverage rule below to each party in turn. 0 when every party is covered
- first_gap_party (string) -- the name of the FIRST party in document order that is not covered. null when parties_uncovered is 0 -- return null rather than an empty string or a guess
- first_gap_reason (enum) one of: no_waiver_on_file, notice_after_waiver, period_short, amount_short, conditional_stale, none -- why the FIRST uncovered party is not covered, decided by the coverage rule in this order and reporting the FIRST condition that is true. 'no_waiver_on_file': there is no waiver on file for that party. 'notice_after_waiver': the party's preliminary notice is dated AFTER the waiver was signed, so the waiver predates the claim -- this outranks everything below it even when the waiver is unconditional, for the full amount and covers the whole period. 'period_short': the party has a PROGRESS waiver whose 'covers work through' date is earlier than the pay application's Period Through date (a FINAL waiver has no through-date and can never be period_short). 'amount_short': the waiver amount is less than that party's amount due this application. 'conditional_stale': the waiver is CONDITIONAL and something has already cleared against it -- either Prior Payment Cleared is yes, or that party is on a joint check arrangement, which clears on issue. 'none' when parties_uncovered is 0

Return a JSON object with exactly these keys: package_id, project_name, pay_app_number, period_through, payment_amount_usd, prior_payment_cleared, release_status, coordinator_note, parties_uncovered, first_gap_party, first_gap_reason
Use null for any field the package does not state.

PROGRESS PAYMENT PACKAGE
------------------------
Package
-------
PA-2027-188

Project
-------
Cedar Point Transit Center

Pay Application
---------------
Application 08

Period Through
--------------
2027-01-31

Payment Amount
--------------
316,504.01 USD

Prior Payment Cleared
---------------------
no

Release Status
--------------
scheduled

Waiver Coverage
---------------
Party 1: Marchand Roofing
  Tier: 1
  Amount due this application: 151,283.50 USD
  Preliminary notice on file: none
  Waiver on file: LW-97184, conditional final waiver
  Waiver amount: 151,283.50 USD
  Waiver covers work through: n/a (final waiver, all work through completion)
  Waiver signed: 2027-02-09
  Joint check arrangement: no

Party 2: Linden Sheet Metal
  Tier: 2
  Amount due this application: 42,231.37 USD
  Preliminary notice on file: 2027-01-17
  Waiver on file: LW-61083, unconditional final waiver
  Waiver amount: 42,231.37 USD
  Waiver covers work through: n/a (final waiver, all work through completion)
  Waiver signed: 2026-12-19
  Joint check arrangement: no

Party 3: Truxton Hoisting
  Tier: 2
  Amount due this application: 70,440.13 USD
  Preliminary notice on file: 2026-11-11
  Waiver on file: LW-15660, conditional final waiver
  Waiver amount: 70,440.13 USD
  Waiver covers work through: n/a (final waiver, all work through completion)
  Waiver signed: 2027-02-05
  Joint check arrangement: no

Party 4: Stanton Fixtures
  Tier: 2
  Amount due this application: 52,549.01 USD
  Preliminary notice on file: none
  Waiver on file: LW-29805, conditional progress waiver
  Waiver amount: 53,049.01 USD
  Waiver covers work through: 2027-02-15
  Waiver signed: 2027-02-10
  Joint check arrangement: yes

Coordinator Note
----------------
Waiver file reviewed at intake and it looked complete.

Raw responseThe raw response

{
  "package_id": "PA-2027-188",
  "project_name": "Cedar Point Transit Center",
  "pay_app_number": "Application 08",
  "period_through": "2027-01-31",
  "payment_amount_usd": 316504.01,
  "prior_payment_cleared": "no",
  "release_status": "scheduled",
  "coordinator_note": "Waiver file reviewed at intake and it looked complete.",
  "parties_uncovered": 2,
  "first_gap_party": "Linden Sheet Metal",
  "first_gap_reason": "notice_after_waiver"
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check lien waivers before a construction progress payment goes out — 55 payment packages. Two tiers of one model family answered, and every answer was then graded Four different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran r001, r002 and r003 publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page. Both tiers are priced against the SAME published card, which is what makes the 15 pct premium a statement about token counts rather than about anybody's list price.

The gradersFour ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The coordinator-note floor (evals/baseline.py) and the scorer (evals/judge.py) are both pure code and cost $0.00 to run against any result set. The figure above is all three runs' own token counts (54, 55 and 55 packages, two tiers) priced at Google Gemini 3 Flash's published rate — the same basis cost_per_query_usd uses, not a second, larger spend. ⚠︎ IT IS ABOUT DOUBLE WHAT THE SIBLING EXTRACTION KITS IN THIS SERIES PROJECT PER RUN, and the reason is on the page rather than hidden: output tokens. A reply here is four times longer than a sibling kit's for a record of the same shape, because the model works the coverage rule through every party before answering.

Cost driversWhat actually moves the bill

• OUTPUT, not input, and that is unusual for an extraction kit. 1,021 output tokens a package against 2,060 in, priced at six times the rate, so output is about 75 pct of the bill. The reply is mostly the model working the five-condition rule through four or five parties before it answers — an eleven-key JSON record is roughly 150 tokens of it.
• The fixed system prompt and field schema (1,456 of 2,049 tokens on the example call, 71 pct) outweigh the package sections sent (593 tokens) — the floor every call pays before a single party is read. It is long because the rule is: five conditions, a priority order, and the two orderings a reader gets backwards spelled out.

Your volumeWhat it costs at your volume

Linear in packages: each call is independent, carries the same fixed prompt and shares nothing with its neighbours, so 550 packages cost ten times 55 and take ten times as long. Nothing amortises — there is no index to build and no cache, which is also why there is no volume discount to find without changing the design. What does NOT scale linearly is the reply: a package with ten parties instead of five will cost more per package than one with five, because the working is per party.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 55 progress-payment packages, one completion call each, on the FAST tier, of which 54 returned a completion. The fast tier's second run and the deliberating tier's own 55 calls are recorded separately in Cost.cost_by_model and are not projected here.

The codeEvery file, and what it is for

Five modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the payment package into addressable sections, pure code — the party blocks stay together inside one Waiver Coverage section, because the coverage question is asked across parties

# Cut a payment package into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections carry each field, pure code — the three coverage answers are mapped to the party blocks, the period-through date and the prior-payment answer, never to the coordinator's note, and the Prime Contractor and Subcontract Reference sections are mapped by nothing and never sent

You change it to: map fields to your own pay-application export's headings; unmatched falls back to the whole document

# Pick which sections plausibly carry each field. Pure code -- the last deterministic step

COVERAGE_SECTIONS = ["Waiver Coverage", "Period Through", "Prior Payment Cleared"]
SECTION_HINTS = {
def for_field(secs, field):
def plan(secs, fields):

src/prompt.pyprompt

assemble one call for all eleven fields, with the five-condition coverage rule stated in full and in order — a notice dated after the waiver outranks everything below it, a final waiver can never be short on period, period outranks amount, and a joint check clears on issue

# Assemble the extraction prompt. One prompt per lien-waiver payment package, all eleven fields

SYSTEM = (
def field_schema(fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key — plus the pure-code business-condition check downstream (a package with a gap that is still scheduled to go out is pulled back) and a no-gold self-check beside it

You change it to: the coverage rule itself — your own counsel's requirements and your own jurisdiction's waiver types. It is stated once and read by the corpus generator, the prompt and the label checker, so changing it here changes all three together

# Extract one progress-payment package's fields: segment, select, prompt, one model call, then a

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 3000
WAIVER_TYPES = ("conditional_progress", "unconditional_progress",
GAP_REASONS = ("no_waiver_on_file", "notice_after_waiver", "period_short", "amount_short",
def load_fields():
def load_doc(pkg_id):
def documents():

evals/judge.pyjudge

score field accuracy, the coverage-gap confusion matrix, gap attribution and the hold flag separately, pure code

# Score an extraction run. PURE CODE -- gold is exact and the answer is one value per cell, so

def norm(v):
def _num(v):
def equal(field, got, want):
def score(fields, records, golds):
def _matrix(rows, positive):
def _yesno(n):
def score_flags(records, flags, checks, golds):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2059 input and 1021 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's payment packages are entirely synthetic (tools/build_corpus.py, seed 20260822): no real project, contractor, supplier, waiver, notice or person exists in the corpus, and nothing was fetched from anywhere. The only outbound traffic the kit makes is one chat-completion request per package to the configured provider, carrying the mapped sections of one package. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env, this kit's own .env (which holds a MODEL line and nothing else) or the real environment, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser.

The experimentWe did not attack it — and three of four boundaries hold

The three boundaries that hold were confirmed by reading the code, not by a run: no code path releases, withholds, requests or files anything; the routing rule and the coverage rule read only counts, enums, amounts and dates, so the coordinator's note cannot reach either; and the key and base URL are redacted out of any error before it is returned. The fourth is open. An indirect prompt injection needs a field an outside party authored, and this kit has two — the coordinator's note and every party name — which makes it a wider opening than the sibling kits in this series carry, and the reason the gap is named rather than glossed. Confirmed by reading the code, not by a run, on 2026-08-22 -- no attack run was fired; see redteam.why.

The first three boundaries hold, confirmed by reading the code rather than by an attack run. The fourth is open and is named as open — this kit has TWO fields an outside party authored (the coordinator's note and every party name), the model is asked to copy both back verbatim, and one of them is the field the attribution grade is scored on. Nothing here has tried either.

The result0 attack trials, and three of four boundaries checked here hold on evidence. The one that does not is the one this kit's own shape invites twice over: the coordinator's note and every party name are both text an outside party wrote, the model is asked to copy both back, and whether an instruction hidden in either could move the coverage answer is unmeasured.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted ambiguity is a REGISTER mismatch — a settled-sounding note on a package with a real gap — which measures whether a model runs the rule when the prose points the other way. It does not measure whether a model obeys an instruction hidden in the same field, and it says nothing at all about a party name chosen to be confusable with another party on the same package. Those are different failures and none of them is measured here.

HonestyWhat this does not prove

• Whether an indirect prompt injection in the coordinator_note field could move parties_uncovered on this model. No attack run exists.
• Whether a crafted PARTY NAME could do the same — a name written to read as an instruction, or one chosen to be confusably similar to another party on the same package. This kit does no name matching at all, so two near-identical names are two parties to it, and nobody has tested what that does to the attribution grade.
• Whether the kit behaves safely against a hostile provider — a response body crafted to break the JSON extraction in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed document, and nothing beyond that was tested.
• Whether a real waiver file would carry anything sensitive this kit mishandles. The corpus has no personal data by construction; a real package carries named companies, contract values, a project address and sometimes an individual signer's name, and none of that path is exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_hold fires when at least one party on the package is uncovered AND the pay application is still scheduled to be released this cycle — parties_uncovered > 0 and release_status == "scheduled". Both values come out of the same reply; the rule is run afterwards in pure code, over whatever the model returned, and never over gold. A reply missing either value returns None rather than False: an unknown is not a pass. IT ROUTES, IT DOES NOT AUTHORISE — and a package with no gap is not thereby cleared for release, it is a package this kit found nothing to say about.

src/extract.py::compute(), called by extract() on every package, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by identical code. The coverage rule it reads lives in a SEPARATE function, coverage_status(), which is also what the corpus generator wrote gold with, what src/prompt.py and data/fields.json state to the model, and what evals/check_labels.py asserts the four sharp cases against — one definition, four readers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a check on whether the extracted values are right. If the model applies the coverage rule wrongly and then reports that wrong answer consistently, the package is routed — or not routed — on a wrong count, and this rule cannot tell. The no-gold self_check beside it is the closest thing to that check and it is reported separately; it passed cleanly on the ONE wrong reply this kit has on record, which is the sharpest demonstration available that it is not a substitute for labels.
• It is NOT any general contractor's release policy and it is NOT a legal test. A-gap-and-still-scheduled is this kit's own simplification, invented for this corpus. No jurisdiction's statute, no filed subcontract and no real payment procedure was consulted, and none is reproduced. A real payment desk weighs the dollar size of the exposure, how far down the tiers the gap sits, whether a bond covers it, and what the subcontract says about withholding.
• It is NOT a determination of anybody's lien rights, and it is NOT a disposition. Nothing here releases or withholds a payment, requests or issues a waiver, or files or answers a notice. parties_uncovered is an extracted answer and needs_hold is a routing signal; a person decides what happens to the money.

WatchedWhat is watched, and why that one

The kit measures 21 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 14 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 55, extraction_cells 605, failures 0, refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 4 runs measured the same system — they differ on documents, extraction_cells, failures, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() or coverage_status() in src/extract.py, on any change to data/fields.json's allowed values for first_gap_reason or release_status, and on any corpus regeneration. Changing WHO GETS PULLED BACK is a policy change and must be re-scored, even though it never changes what covered means — and changing the coverage rule's priority ORDER alone moves every attribution while moving no verdict, so attribution must be re-scored even when the matrices do not move.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per package — segment, select, prompt, one call, parse, compute — with no branch, no loop and no agent. The only fan-out in the whole kit is inside the model's own reply, where it works five conditions across four or five parties, and that is not orchestration, it is a paragraph.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain: the JSON extraction from a fenced reply, the retry/backoff policy, the section regex, the number normaliser and the .env reader are all code somebody has to own — and two of those five shipped a defect this kit caught and recorded.
• No framework means no framework's ecosystem — no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. What ships is what is in the repository.
• The provider abstraction covers exactly two wire formats, and its retry policy covers exactly one class of failure. A third provider is one function and one dict entry; transport-level retry is four lines and is on the add_first list rather than in the code, because both published runs were made without it.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-waiver-match on the fast tier, run again, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

1 run not plotted. b000-waiver-match-rules recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Every project, contractor, supplier, waiver number, amount, date and coordinator note is invented; no real project, company, filed notice or waiver document is named or reproduced, and no jurisdiction's statutory waiver form is reproduced. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-field exact match, light normalisation

Does the model's value for each field match gold, after trimming whitespace and punctuation and treating numbers within half a cent as equal? first_gap_party is the one legitimately-null field — null exactly on the packages where nothing is uncovered — so a null there is a hit, not a miss, when gold agrees.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every call, and the same field-match logic the free baseline is scored by — a baseline and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same waiver types, amounts, dates, joint-check flags and party order the package states; evals/check_labels.py asserts every non-nullable field is populated on every row, that first_gap_party's nullness agrees with parties_uncovered, that every coverage answer agrees with the rule run over its own values, and that all four sharp cases of the rule compute correctly, before any run is allowed to spend.

How tight can the band be? There is no tolerance band on the field grade itself — it is exact match after trimming whitespace and punctuation, and numbers within half a cent are treated as equal, never a continuous score to round.

Cadence: Re-score on any change to src/prompt.py, data/fields.json or tools/build_corpus.py — the first two change what is asked, the third changes what is asked about. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineIs any party on this package uncovered, against gold's own rule

Of every package that really has an uncovered party, how many did the run say has one — and how many fully covered packages did it wrongly flag? A GAP IS THE POSITIVE CLASS: a package with an uncovered party that gets called complete is the failure a payment desk actually pays for, because the money goes out.

MethodHow the test was run

evals/judge.py::score_flags, in-process, no key and no model.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's count is coverage_status() applied party by party in document order, the same five conditions in the same priority order the kit publishes and states to the model — so the truth this matrix grades against can never be a separately-typed label that drifted from the rule.

How tight can the band be? No threshold — the verdict is derived from a count, and a reply that returns no usable count is recorded as unanswered rather than folded into the correct-negative cell.

Cadence: Re-run whenever coverage_status() in src/extract.py changes, whenever the corpus is regenerated, and on any provider or model change.

The decisionWhen to reach for it

In one lineWhich party, and why — scored only where there is a gap

On the packages that really do have a gap, did the run name the right party AND the right reason? A package-level yes/no hides the difference between finding THE gap and finding A gap, and on a package with four parties and five possible reasons that difference is the entire value of the thing — the person reading the flag goes and opens the party it names.

MethodHow the test was run

evals/judge.py::score_flags, in-process, no key and no model. Scored only on the 31 packages gold says really have a gap — a fully covered package has no party and no reason to attribute, and padding the denominator with rows nobody can get wrong would make every figure here look better than it is.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's first_gap_party and first_gap_reason, both read off coverage_status() applied in document order — the first party for which any condition fires, and that party's own first-firing condition. Never a typed label.

How tight can the band be? No threshold — party is exact string match after trimming and case-folding, reason is one of six allowed values. The denominator is stated beside the rate because 31 rows makes every rate wide.

Cadence: Re-run whenever coverage_status()'s priority ORDER changes, even if no condition's definition changes — reordering moves which reason is reported without moving any verdict.

The decisionWhen to reach for it

In one lineneeds_hold against the same rule run over gold

Does the pure-code routing decision — a gap AND still scheduled for release — land on the same packages it would land on if both fields had been read perfectly? It is a BUSINESS CONDITION, so unlike a self-consistency check it genuinely needs labels, and saying so is half of what makes the number believable.

MethodHow the test was run

evals/judge.py::score_flags, in-process, no key and no model.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute(), the same function the run uses, applied to GOLD's parties_uncovered and release_status. One rule, two inputs, so a change to the rule moves both sides together and the grader cannot silently grade an old policy.

How tight can the band be? No threshold — a count compared against zero, an enum compared against one value, and an AND.

Cadence: Re-run whenever compute() changes. Changing WHO GETS PULLED BACK is a policy change and must be re-scored, even though it never changes what covered means.

The decisionWhen to reach for it