The business caseThe problem this solves

A construction payment gets held because a compliance item is open — an insurance certificate, a certified payroll return, a lien waiver for the last cycle. Every one of those flags was set on a date by somebody, and nothing on the file says how long that decision stays worth trusting. So holds accumulate on reasoning nobody has revisited: a window that elapsed months ago, a flag nobody ever set, a citation pointing at a record that is not on the file, and the expensive one — a flag read off the folder BEFORE the document it was waiting for arrived. The payment stays stopped and the paperwork is already there. Somebody opening one payee's payment file, reading the hold notice, finding each compliance item it names, and then deciding — from memory, per item class and per project — whether the date beside the flag is recent enough to still mean anything. The part that goes wrong is not the reading. It is that a fresh-looking date can sit above a document that arrived after it.

The inputThe actual payment hold file

The corpus is 50 payment hold file, 0.11 MB (txt 50). Plain text, one format, invented rather than fetched — a real payment hold file states a real business's position on money it is owed and has not been paid, which is commercially sensitive on one side and legally consequential on the other. And there is a second reason that is stronger here: HALF THE ANSWER IS NOT IN THE DOCUMENT. Labelling a captured corpus would mean somebody deciding by hand how stale is too stale for each requirement, which is a commercial and legal judgement rather than a label. Generating it makes gold mechanical instead — decide_item() over the same dates the file prints.

Synthetic Record
----------------
This is a SYNTHETIC payment hold file, generated for a public AI use-case kit. Every payee,
project, pay application, compliance-item reference, subcontract reference, remittance
reference, contact and date in it is invented. No real contractor, owner, project, payment,
compliance requirement, statute or contract form is named or reproduced. The verification
currency policy this file is read against is the kit's own construction and is not an
authority; it asserts no hold-duration limit and no prompt-payment rule.

Pay Application
---------------
PA-2027-0143

Payee
-----
Marchmont Drywall (PAY-3301)
Subcontract: SC-4400

Project
-------
Marlow Quay Cold Store
Verification regime: standard

Cycle
-----
Cycle 2027-03, file drawn 2027-03-16

Payment Status
--------------
State: held
Hold set on: 2027-03-04
Released on: not released
Release decided by: nothing has been released

Hold Notice
-----------
Payment is held pending these compliance items: CI-70106, CI-70107

Compliance Item CI-70107
------------------------
Class: certified_payroll
Requirement: SC-4400, compliance schedule
Flag: expired -- the document on file has run out of its own validity, so it is no longer satisfied
Last verified: 2026-12-19
Evidence received: 2026-11-05

Compliance Item CI-70106
------------------------
Class: lien_waiver_prior_cycle
Requirement: SC-4400, compliance schedule
Flag: deficient -- a document is on file and it does not meet the requirement; it is not expired
Last verified: 2027-02-12
Evidence received: nothing has been received

Compliance Item CI-70108
------------------------
Class: w9_tax_form
Requirement: SC-4400, compliance schedule
Flag: not_received -- nothing has been logged against this requirement
Last verified: 2026-11-09

The outcomeWhat a good result looks like

A row per compliance item the hold notice names, with the currency of its flag as of the day the hold was set, the date it stopped being current, how many days ago that was, and the shipped policy's own reasoning — plus one payment-level answer: is at least one reason behind this hold still current.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• A pay-application cycle where holds are set against named compliance items and each item carries both a verification date and a document-log date — this kit, with data/policy.json replaced by your own refresh windows
  The whole finding is the comparison between those two dates against a window neither of them knows about. Where both columns exist, the free floor scores 48.72 pct on stale flags and 0 of 12 on the superseded ones; the pipeline scores 100 pct on both.
• A compliance register with one date per item and no separate evidence log — the free floor in evals/baseline.py, or a spreadsheet
  With one column there is no superseded finding to make, and what remains is a window comparison a spreadsheet does for nothing. The floor already reads every printed field on this corpus exactly right — 985 of 985 cells — so paying a provider to do the reading buys nothing.
• Deciding whether a hold has run too long, or what an improper hold costs — counsel and the contract — not this kit and not any part of it
  Prompt payment is governed by law and by contract, both varying by jurisdiction and by job. This kit researched neither, so it asserts no hold-duration limit and no prompt-payment rule anywhere: not in data/policy.json, not on the UI, not on these pages.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-23. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-23 — r001-hold-reason. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m evals.check_labels re-derives all 70 cited rows from the shipped policy, audits the privacy of the selection over all 50 files and proves the free floor's two structural zeros — free, no key, no network. python -m evals.run --run-id t000-hold-reason-stub --stub drives the whole pipeline against a regex stub. python -m src.app renders and stays browsable with no key; the button returns a plain sentence saying nothing was called.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. Wall-clock per payment file across 50 files on r001-hold-reason, 12 concurrent workers, one completion call each. p50 5824 ms, p95 10649 ms, whole run 29.0 s.

Current processWhat it replaces

Somebody opening one payee's payment file, reading the hold notice, finding each compliance item it names, and then deciding — from memory, per item class and per project — whether the date beside the flag is recent enough to still mean anything. The part that goes wrong is not the reading. It is that a fresh-looking date can sit above a document that arrived after it.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on the underlined section headings each file prints — the payment status, the hold notice, and one section per compliance item block. No retrieval index is built and nothing is embedded.

SetupWhat the setup figure measured

There is no index. The figure is the cost of segmenting all 50 files in pure Python — nothing is retrieved, because every call reads exactly one file.

LicenceLicence

MIT, this repository's own licence, covering the corpus as well as the code

Bring your ownBring your own payment hold file

REPLACE data/policy.json FIRST. It is this kit's own construct, not an authority, and every currency verdict on every page is its opinion. Then data/corpus/*.txt, data/fields.json, and a gold record per payment. SECTION_HINTS and the `Compliance Item ` prefix in src/select.py map fields to headings and will need editing for a different layout; when they do not match, selection falls back to the whole file minus NEVER_SENT — slower, more expensive, always correct. compute() in src/extract.py is the escalation and is the second thing to replace.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 3,916 tokens assembled, 1,312 are contexts — 34% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method — four calls at max_tokens=1 on HOLD-0020, each prompt the previous plus exactly one more part, every count taken from the provider's own prompt_tokens field rather than estimated. results/tokens-p001-hold-reason.json.

Full promptThe prompt in full

You read a PAYMENT HOLD FILE -- the record of one construction pay application, the compliance items logged against that payee, and the hold notice if one was raised -- and you report what is in it. You return JSON and nothing else.

YOU DO NOT DECIDE WHETHER ANY FLAG IS STALE, AND YOU DO NOT DECIDE WHETHER THE HOLD SHOULD STAND. A separate piece of code works both of those out afterwards: it looks up how many days a verification of each item class stays current under this project's regime, compares that against the dates you report, and joins the hold notice's references against the item blocks you return. Your job is to report the file faithfully, including the places where it records nothing.

DO NOT NAME AN ITEM THE FILE DOES NOT CARRY A BLOCK FOR. Whether a cited reference has a record is decided by the code downstream; an item you add because the hold notice mentions it is an invented record, and it is scored as one.

RULES, in order of importance:
1. `last_verified` AND `evidence_received` ARE TWO SEPARATE LINES AND MUST BE REPORTED SEPARATELY. `Last verified` is the day somebody last checked the FLAG. `Evidence received` is the day the document behind the requirement was logged. They are frequently different dates and the comparison between them is the single most important thing on the page. Never copy one into the other, and never report the later of the two because it looks more current.
2. `hold_cited_refs` IS EVERY REFERENCE PRINTED ON THE HOLD NOTICE LINE, comma-separated, in the order the line prints them, verbatim. That line is the denominator of the whole review. A reference dropped from it is a reason for the hold that disappears entirely, and downstream it is indistinguishable from a reason nobody ever recorded. Where the file carries no Hold Notice section at all, return null -- which happens, and is a real answer meaning no hold was raised.
3. RETURN ONE ENTRY PER COMPLIANCE ITEM BLOCK ON THE FILE, in the order the file prints them, INCLUDING blocks the hold notice does not name and INCLUDING blocks whose flag says 'satisfied' or 'waived'. A cited item whose flag is satisfied is a payment held against a requirement the file records as met; an uncited item is an open item that is not holding this payment. Those are different findings and dropping either makes them indistinguishable.
4. `flag_status` IS THE FIRST WORD OF THE 'Flag' LINE AND NOTHING ELSE. The explanation after the dash can be long and can mention other states; it does not change the flag. Report the flag exactly as it stands.
5. `item_class` IS COPIED VERBATIM FROM THE 'Class' LINE. It is the key the refresh window is looked up on downstream, so an approximation is not an approximation: an item with no window is reported as having no record at all.
6. A DATE IS A DATE OR IT IS NULL. Where a line says no verification is recorded, or that nothing has been received, or that no hold was set, or that nothing has been released, return null -- not today's date, not the cycle date, not a date from another line and not the date the file was drawn.
7. `payment_state` AND `release_decided_by` ARE READ FROM THE 'State' AND 'Release decided by' LINES. Report the role recorded; you are not being asked whether that role was entitled to decide, and nothing you return decides anything.
8. `verification_regime` IS COPIED IN FULL, verbatim. It selects which refresh windows apply to this project, so a shortened or abbreviated form is a different question being answered.
9. Copy identifiers verbatim. Write every date as YYYY-MM-DD, exactly as the file does. Use the exact allowed value for a field that lists them, and return every field named in the schema on every entry, even when the answer is null.

VERIFICATION CURRENCY POLICY (the authority for the currency column; this is an ILLUSTRATIVE policy written for this kit, and it reproduces no statute, prompt-payment law, contract form, owner's payment procedure or contractor's compliance manual)

WHAT THE REVIEW WORKLIST IS
  A row per compliance item the hold notice names, each carrying the item's flag, the date somebody last verified that flag, and THE CURRENCY OF THE FLAG when the hold was set: current, stale because the refresh window elapsed, stale because the evidence arrived after the last check, never verified at all, cited with no record on the file, or not a hold reason in the first place. It is a review worklist. It releases nothing, holds nothing, pays nothing and decides nothing.

THE HARD HALF
  The document tells you the flag and the date. It does not tell you HOW LONG a verification of this item class, under this project's verification regime, stays worth trusting. That window lives in this file, outside the document, and it is what turns a printed date into an answer. A flat staleness window is wrong in both directions at once: this policy's windows run from 7 days to 730, so one number both clears items that went stale weeks ago and raises items nobody needed to look at.

VERIFICATION REGIMES
  standard         a private commercial project with no statutory payroll reporting and no payment bond -- the base windows apply unchanged
  public_works     a publicly funded project whose payroll and insurance evidence is re-checked far more often; certified payroll drops to a weekly window and insurance to three weeks
  bonded_private   a private project carrying a payment bond, where the bond rider is re-checked monthly rather than quarterly

COMPLIANCE ITEM CLASSES
  certificate_of_insurance     certificate of insurance  [hold-critical]
  subcontractor_licence        trade licence or registration  [hold-critical]
  bond_rider                   payment bond rider  [hold-critical]
  certified_payroll            certified payroll return
  lien_waiver_prior_cycle      lien waiver for the prior cycle
  safety_orientation_record    site safety orientation record
  w9_tax_form                  taxpayer identification form
  preliminary_notice_ack       preliminary notice acknowledgement

REFRESH WINDOWS, in days -- THE ITEM CLASS'S AND THE REGIME'S, not the payment's
  How many days a verification of this item class stays trustworthy. It is a property of THE ITEM CLASS and THE PROJECT'S REGIME, never of the payment -- which is exactly what one flat staleness window cannot express. Every number here was invented for this kit.

  default:
    certified_payroll               14
    certificate_of_insurance        30
    lien_waiver_prior_cycle         45
    preliminary_notice_ack          60
    bond_rider                      90
    safety_orientation_record      180
    subcontractor_licence          365
    w9_tax_form                    730
  public_works overrides: certified_payroll 7, certificate_of_insurance 21
  bonded_private overrides: bond_rider 30

FLAG STATUSES
  satisfied      the requirement is met on the file. An item whose flag says this is not a reason to hold anything
  deficient      a document is on the file and it does not meet the requirement
  expired        a document is on the file and its own validity has run out
  not_received   nothing has been logged against this requirement
  waived         the requirement has been waived in writing for this payee on this project

THE CURRENCY OF A FLAG, WORKED OUT AFTERWARDS IN CODE (context for you; you do not compute it)
  1 not_a_hold_reason -- the hold notice names this item and the item's own flag says `satisfied` or `waived`. The flag is perfectly current and it says nothing is wrong; the payment is being held against a cleared requirement. Checked FIRST, because asking whether such a flag is stale is asking the wrong question about it.
  2 no_record_on_file -- the hold notice names a reference the file carries no compliance-item block for. Neither current nor stale: there is nothing to be stale. Counted as current it clears a hole; counted as stale it sends somebody to refresh a record that does not exist.
  3 never_verified -- the item block records no verification date at all. Nobody has ever checked this flag, so the hold has never rested on anything. Different from elapsed, and a different action.
  4 superseded_by_evidence -- the evidence behind the item was received AFTER the last verification. The flag predates the paperwork. Checked BEFORE the window, because a flag inside its window is still worthless if the document arrived after it was read.
  5 stale_elapsed -- more days have passed between the last verification and the date the hold was set than this item class's refresh window allows under this project's regime.
  6 current -- verified inside the window, with no later evidence on the file. This is the only value that makes a hold well-founded.

  A hold is WELL-FOUNDED when at least one item the hold notice names comes back `current`. Nothing else on this page is a judgement about whether the payment should be held -- that is a person's decision and this policy does not make it.

WHY `superseded_by_evidence` IS FIRST CLASS
  A flag can be well inside its refresh window and still be worthless. If the evidence behind an item arrived AFTER the last verification, the verification was performed on a file that did not yet contain it -- so `not_received` is a reading of a folder that has since changed. This is the stale hold that a window-only reader clears: the date is fresh, the answer is wrong, and a legitimate payment stays blocked while the paperwork sits on the file. It is checked BEFORE the elapsed window on purpose, because the two have different actions: elapsed means go and verify; superseded means the document is already there and nobody looked.

WHAT THIS PRODUCES, AND WHAT IT DOES NOT
  Nothing here decides anything. The project manager sets and releases holds; this kit reports what the file says and how old the reasoning behind it is.

Return these:
- application_ref (string) -- the pay application reference printed in the Pay Application section, verbatim (for example PA-2027-0143)
- payee_name (string) -- the payee's name in the Payee section, verbatim and WITHOUT the bracketed payee reference after it
- verification_regime (enum) one of: standard, public_works, bonded_private -- the regime on the 'Verification regime' line of the Project section, verbatim. It decides HOW LONG each kind of verification stays current on this project, so a near-miss is a different question being answered
- payment_state (enum) one of: held, released, not_held -- the FIRST WORD of the 'State' line in the Payment Status section. 'held' means the payment is still being withheld today, 'released' means a hold was set and has since been lifted, 'not_held' means no hold was ever set on this application
- hold_set_on (date) -- the date on the 'Hold set on' line, as YYYY-MM-DD. EVERY currency judgement on this payment is counted from it. Return null where the line says no hold was set
- released_on (date) -- the date on the 'Released on' line, as YYYY-MM-DD. Return null where the line says the payment has not been released -- do not substitute the hold date or the date the file was drawn
- release_decided_by (enum) one of: project_manager, contracts_manager, payment_clerk, automated_workflow -- the role on the 'Release decided by' line, verbatim. Return null where nothing has been released. This kit REPORTS who is recorded as having decided; it never decides, and it does not say who was entitled to
- hold_cited_refs (string) -- EVERY compliance-item reference printed on the Hold Notice line, comma-separated, in the order the line prints them, verbatim (for example 'CI-70142, CI-70147'). Return null where the file carries no Hold Notice section at all. A reference dropped here is a reason for the hold that disappears from the review entirely, which is the most expensive single mistake available on this shape
- items (array of objects) -- ONE OBJECT PER COMPLIANCE ITEM BLOCK printed on the file, in printed order, cited and uncited alike, each carrying:
    - item_class (enum) one of: certificate_of_insurance, subcontractor_licence, bond_rider, certified_payroll, lien_waiver_prior_cycle, safety_orientation_record, w9_tax_form, preliminary_notice_ack -- the class stated on the 'Class' line of the item block, verbatim. THIS IS THE KEY the refresh window is looked up on
    - item_ref (string) -- the reference on the item block heading, verbatim (for example CI-70142)
    - flag_status (enum) one of: satisfied, deficient, expired, not_received, waived -- the FIRST WORD of the 'Flag' line and nothing else. The words after the dash explain the flag and do not change it, and a 'satisfied' or 'waived' item is still reported when the hold notice names it -- that is a finding, not a row to drop
    - last_verified (date) -- the date on the 'Last verified' line, as YYYY-MM-DD -- the day somebody last checked THE FLAG, which is not the day any document was issued or received. Return null where the line says no verification is recorded on the file
    - evidence_received (date) -- the date on the 'Evidence received' line, as YYYY-MM-DD -- the day the document behind this requirement was logged. Return null where the line says nothing has been received. It is a SEPARATE line from 'Last verified' and the two are frequently different dates; copying one into the other destroys the only comparison on this page that matters

Return a JSON object with exactly these top-level keys: application_ref, payee_name, verification_regime, payment_state, hold_set_on, released_on, release_decided_by, hold_cited_refs, items
`items` is an array. Return it EMPTY only if the file carries no compliance item block at all -- which happens, and is a real answer.

PAYMENT HOLD FILE
-----------------
Synthetic Record
----------------
This is a SYNTHETIC payment hold file, generated for a public AI use-case kit. Every payee,
project, pay application, compliance-item reference, subcontract reference, remittance
reference, contact and date in it is invented. No real contractor, owner, project, payment,
compliance requirement, statute or contract form is named or reproduced. The verification
currency policy this file is read against is the kit's own construction and is not an
authority; it asserts no hold-duration limit and no prompt-payment rule.

Pay Application
---------------
PA-2027-0190

Payee
-----
Trevanion Piling (PAY-3320)
Subcontract: SC-4532

Project
-------
Northwold Teaching Block
Verification regime: bonded_private

Cycle
-----
Cycle 2027-03, file drawn 2027-03-24

Payment Status
--------------
State: held
Hold set on: 2027-03-08
Released on: not released
Release decided by: nothing has been released

Hold Notice
-----------
Payment is held pending these compliance items: CI-70260, CI-70264, CI-70269

Compliance Item CI-70277
------------------------
Class: preliminary_notice_ack
Requirement: SC-4532, compliance schedule
Flag: satisfied -- the requirement is met on this file
Last verified: 2027-02-16
Evidence received: 2027-02-10

Compliance Item CI-70269
------------------------
Class: certified_payroll
Requirement: SC-4532, compliance schedule
Flag: not_received -- nothing has been logged against this requirement
Last verified: no verification is recorded on the file
Evidence received: nothing has been received

Compliance Item CI-70264
------------------------
Class: certificate_of_insurance
Requirement: SC-4532, compliance schedule
Flag: expired -- the document on file has run out of its own validity, so it is no longer satisfied
Last verified: 2026-10-06
Evidence received: 2026-09-17

Compliance Item CI-70272
------------------------
Class: w9_tax_form
Requirement: SC-4532, compliance schedule
Flag: deficient -- a document is on file and it does not meet the requirement; it is not expired
Last verified: 2026-09-04
Evidence received: 2026-08-31

Compliance Item CI-70260
------------------------
Class: safety_orientation_record
Requirement: SC-4532, compliance schedule
Flag: deficient -- a document is on file and it does not meet the requirement; it is not expired
Last verified: 2027-02-23
Evidence received: 2027-03-01

Compliance Item CI-70280
------------------------
Class: subcontractor_licence
Requirement: SC-4532, compliance schedule
Flag: deficient -- a document is on file and it does not meet the requirement; it is not expired
Last verified: 2026-11-30
Evidence received: 2026-11-20

File Notes
----------
Compliance items are printed in the order they were logged, cleared items included.
A hold notice names the items the hold was raised against and nothing else.
The Flag line records the last compliance decision taken on an item. The Evidence
line records the document log. The two are maintained by different processes and a
document arriving does not update the flag.

Raw responseThe raw response

{
  "application_ref": "PA-2027-0190",
  "payee_name": "Trevanion Piling",
  "verification_regime": "bonded_private",
  "payment_state": "held",
  "hold_set_on": "2027-03-08",
  "released_on": null,
  "release_decided_by": null,
  "hold_cited_refs": "CI-70260, CI-70264, CI-70269",
  "items": [
    {
      "item_class": "preliminary_notice_ack",
      "item_ref": "CI-70277",
      "flag_status": "satisfied",
      "last_verified": "2027-02-16",
      "evidence_received": "2027-02-10"
    },
    {
      "item_class": "certified_payroll",
      "item_ref": "CI-70269",
      "flag_status": "not_received",
      "last_verified": null,
      "evidence_received": null
    },
    {
      "item_class": "certificate_of_insurance",
      "item_ref": "CI-70264",
      "flag_status": "expired",
      "last_verified": "2026-10-06",
      "evidence_received": "2026-09-17"
    },
    {
      "item_class": "w9_tax_form",
      "item_ref": "CI-70272",
      "flag_status": "deficient",
      "last_verified": "2026-09-04",
      "evidence_received": "2026-08-31"
    },
    {
      "item_class": "safety_orientation_record",
      "item_ref": "CI-70260",
      "flag_status": "deficient",
      "last_verified": "2027-02-23",
      "evidence_received": "2027-03-01"
    },
    {
      "item_class": "subcontractor_licence",
      "item_ref": "CI-70280",
      "flag_status": "deficient",
      "last_verified": "2026-11-30",
      "evidence_received": "2026-11-20"
    }
  ]
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check every reason a payment is on hold — 50 payment hold file. One model answered, and every answer was then graded Four different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Run it twiceThe same set, run again

THE BAND IS 100 PCT AND ITS DENOMINATOR IS 4 PCT OF THE CORPUS. Two documents were re-asked, not fifty, and both are among the fullest rather than a random sample — chosen because a longer reply is where a dropped block is most likely, not because they are representative. Nothing here establishes that the 48 files nobody asked twice are stable, and 13 of the 14 payments that escalate are in that 48. On this shape the gap matters: the review is the hold notice's citations MINUS the blocks the reply returned, so a dropped block becomes a confident no_record_on_file that cites nothing, renders identically to a correct answer, and leaves the escalation firing either way.

item blocks returned on the 2 fullest files, asked 3 times each — 11 of 11 every time. NOT the 50-file scored set — The six asks are not averaged into the scored run and must not be: they cover 11 of the corpus's 117 item blocks and were fired to test STABILITY, not accuracy. Folding them into r001-hold-reason would move a 117-block denominator with 11 blocks of evidence.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersFour ways to grade

The floor is the first row of the evals board deliberately. It costs $0.00, it reads all 985 cells exactly right, and it still loses on every verdict — which means the money on this kit buys the COMPARISON, not the reading. And the row underneath it is the uncomfortable one: the free STUB, forty lines of regex with no provider at all, matches the paid run on every published figure. Both belong on the same board.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The flat 30-day floor (evals/baseline.py), the label gate (evals/check_labels.py) and the scorer (evals/judge.py) are all pure code and cost $0.00 to run against any result set. The figure above is the ONE scored run's own token counts (50 files, 182763 input and 40978 output tokens) priced at Google Gemini 3 Flash's published rate — the same basis cost_per_query_usd uses. ⚠︎ IT IS NOT WHAT BUILDING THIS KIT COST. Outside the scored run: a 6-file calibration on the fullest files ($0.0351), the 22-file prompt ablation ($0.0906), the 6-ask repeat-stability probe ($0.0359), 4 prompt-split calls at max_tokens=1 ($0.0051), the worked-example capture ($0.0056), and 3 live UI screenshot calls whose tokens are not separately recorded and are therefore not projected. The shared call ledger records 92 live calls for this kit on 2026-08-23; everything with a committed token count projects to $0.3866 on this card.

Cost driversWhat actually moves the bill

• The fixed prefix. 3082 of the worked example's 3916 input tokens — the system prompt, the whole currency policy and the field schema — are byte-identical on all 50 calls, 78.7 pct of every one.
• Provider-side reasoning. 29646 of the run's 40978 output tokens (72.3 pct) were spent before a character of the review was written, and output is priced at six times input on this card.
• The number of compliance items a payee carries. The reply is one object per item block, so a file with six blocks costs more than a file with one — 117 blocks across 50 files, and the largest reply was 1749 output tokens against a 40000 ceiling.

Your volumeWhat it costs at your volume

Linear in payment files: each call is independent, carries the same fixed prefix and reads exactly one file, so 500 files is 10x $0.004286 and nothing changes shape. What does not scale linearly is the POLICY — it grows with the number of item classes and regimes a contractor runs, not with the number of payments, and it is re-sent on every single call. A contractor with forty item classes renders a far larger block into every one of thousands of calls while the files stay the same size.

Model choiceWhy this model

One tier, one key, one run — the same tier every kit in this series runs first, so the rows compare. No second tier was fired, so nothing on these pages is a model comparison, and the two probes that WERE fired both argue that this corpus could not support one anyway.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 payment hold files, one completion call each, one tier. The calibration, the prompt ablation, the repeat-stability probe, the prompt-split calls and the worked-example capture are recorded separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

Eight modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/app.pyThe local UI

One payment file, one review, on your machine with your key. It holds no pipeline logic: it serves data/policy.json verbatim so the windows the review was decided by are one click from the review itself, and its primary table is the hold notice's citations rather than the file's item list.

# The minimal local UI. Standard library only -- python -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "9007"))
class H(BaseHTTPRequestHandler):
def main():

src/segment.pySegment

Cuts a file into addressable sections on its underlined headings, so every extracted value can name the span it was read from. Pure code, no model. A file with no headings falls back to ONE whole-document section rather than pretending to a carve-up it cannot make.

# Cut a payment hold file into addressable sections. Pure code -- no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pySelect — a swap seam

Maps each field to the sections that can answer it and sends their union. Payee Contact Reference — project accountant, internal extension, the payee's remittance reference, a banking contact — is mapped by nothing and therefore never leaves the machine, including on the ten files that take the fallback.

You change it to: Which parts of a file reach the wire, and what never does. A different file layout needs SECTION_HINTS and the `Compliance Item ` prefix changed; unmatched hints fall back to the whole file minus NEVER_SENT.

# Pick which sections of a payment hold file are sent. Pure code -- the last deterministic step

BANNER = "Synthetic Record"
APP = "Pay Application"
PAYEE = "Payee"
PROJECT = "Project"
CYCLE = "Cycle"
STATUS = "Payment Status"
HOLD = "Hold Notice"
NOTES = "File Notes"
ITEM = "Compliance Item "

src/policy.pyThe currency policy

Loads data/policy.json — the half of the answer that is not in any document: how many days a verification of each item class stays current under each project regime, 7 to 730 — and applies six ordered checks per cited reference. The corpus generator, the prompt and the scorer all read this one copy.

# THE VERIFICATION-CURRENCY POLICY, loaded from data/policy.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
POLICY_PATH = os.path.join(HERE, "data", "policy.json")
FLAG_STATUSES = ("satisfied", "deficient", "expired", "not_received", "waived")
CURRENCY = ("current", "stale_elapsed", "superseded_by_evidence", "never_verified",
STALE = ("stale_elapsed", "superseded_by_evidence", "never_verified")
PAYMENT_STATES = ("held", "released", "not_held")
DECIDERS = ("project_manager", "contracts_manager", "payment_clerk", "automated_workflow")
def load(path=POLICY_PATH):
P = load()

src/prompt.pyPrompt

One call per payment file. The model is asked ONLY what the file records: the flag, the two dates, and the references the hold notice names. The first paragraph tells it that it does not decide whether anything is stale, and rule 1 is that the two date lines are separate and must stay separate.

# Assemble the payment-hold prompt. ONE prompt per payment file, every item it carries in the reply.

SYSTEM = (
POLICY_TEXT = policy_block()
def field_schema(app_fields, item_fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/adapters/__init__.pyThe model call — a swap seam

One interface, two providers behind it, raw HTTP over the standard library so a forker runs it on whichever key they already hold. Transient statuses and dropped connections are retried with backoff; terminal ones are not.

You change it to: PROVIDER and MODEL in .env. Two adapters ship (OpenAI-compatible and Anthropic Messages); adding a third is one function and one dict entry. Nothing else in the kit knows which provider answered.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/extract.pyThe join and the guardrail — a swap seam

Runs the currency join over what the model returned and then compute(), which fires when a payment is still held, the hold names at least one item, and not one of them is current. Never over gold, and returning None rather than False wherever the answer cannot be reached.

You change it to: compute() is one predicate over the review rows and the payment state. Five alternatives were measured over the corpus before this one shipped; the table is printed by tools/build_corpus.py on every generation.

# Read one payment hold file, then do the comparison: segment, select, prompt, one model call,

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 40000
def load_fields():
def item_fields(fields=None):
def app_fields(fields=None):
def load_doc(doc_id):
def documents():

evals/judge.pyThe scorer

Five graders, pure code, no model and no key: the cells, the six-way currency verdict with its false-alarm rate, stale-flag recall with superseded scored separately, the refresh-due date, and the hold verdict per payment. The free floor is scored by the same functions.

# Score a compliance-hold currency run. PURE CODE -- gold is exact and every answer is one value or

APP_CELLS = ("application_ref", "payee_name", "verification_regime", "payment_state",
ITEM_CELLS = ("item_class", "item_ref", "flag_status", "last_verified", "evidence_received")
SPANNABLE = ("item_class", "item_ref", "last_verified", "evidence_received")
NON_SPANNABLE = ("flag_status", "payment_state")
ACTIONABLE = tuple(c for c in PL.CURRENCY if c != "current")
FLAT_WINDOW_DAYS = 30
def norm(v):
def ref_norm(v):
def refs_norm(v):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3655 input and 819 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's payment hold files are entirely synthetic (tools/build_corpus.py, seed 20260823) and no real contractor, owner, project, payment or person is named anywhere in them. The one section that looks like real operational data — Payee Contact Reference, carrying a project accountant, an internal extension, the payee's remittance reference and a banking contact — is mapped by no field in src/select.py and therefore never reaches the wire, on any file, including the ten where the fallback path fires.

Read from the shared .env or the real environment only, never written to a result file, never printed, never logged. src/config.py::sources() reports WHICH file supplied a value and never the value. src/app.py redacts the key and the base URL out of any provider error before it reaches the page.

The experimentWe did not attack it — and the boundary that would matter most here is the one an attacker would not need

Three boundaries hold by construction and were confirmed by reading the code: nothing in the kit moves money, nothing writes a credential to a result file, and the contact section is mapped by no field. The fourth was measured, not read: four files carry a payee's own claim that a document was sent, on items whose blocks record that nothing was received, and the run followed the block on all four. That is the shape that matters on a payments kit — not an adversarial string, a plausible sentence from the party who wants the money. Confirmed by reading the code and by evals/check_labels.py's privacy audit over all 50 files, which runs before any spend, not by an attack run, on 2026-08-23.

Three boundaries hold by construction and one by measurement, and the measured one is the interesting one: the payee's note is the only adversarial text in this corpus and it is there because a real deployment would let an outside party write into that section. It is not a prompt-injection probe and is not presented as one.

The result0 attack trials, and three of four boundaries here hold by construction rather than by measurement. The fourth — an outside party's claim in the file notes — was answered correctly on all four files that carry it.

This run's corpus is entirely generated (tools/build_corpus.py) and no adversarial run was fired against it. What the corpus does carry, by design, is four files whose File Notes hold a claim an outside party wrote — the realistic version of the problem on a payments kit, where the pressure to release comes from the party being paid. Gold follows the record; the run agreed on all four.

HonestyWhat this does not prove

• Whether the pipeline resists a deliberate prompt injection. None was tried, and the File Notes section is sent to the model on purpose so that a future probe has a surface to attack rather than a hidden one.
• Whether the redaction in src/app.py covers every provider's error shape. It replaces the key and the base URL by string match, which was checked by reading the code and against one provider's errors.
• Anything about a deployment. Nothing here is behind a network boundary, an auth layer or an audit trail, and no part of this page claims otherwise.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

The escalation fires when a payment is STILL HELD, the hold notice names at least one compliance item, and NOT ONE of them comes back `current`. It is a business condition, not a check on the model: a payment blocked with nothing behind it that anybody has stood behind recently.

src/extract.py::compute(), called by extract() on every file and re-run by evals/judge.py::score_flag over the run's OWN rows as a no-gold diagnostic

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is not a decision to release or withhold anything. Nothing in this kit moves money, clears a compliance item or notifies anybody; the project manager does all of that.
• It is not a statement that the hold is wrong. A payment can be perfectly properly held on a requirement that is genuinely open and whose flag simply has not been re-checked. What the condition says is that nobody has stood behind any of the reasons recently.
• It is not a hold-duration rule. No limit on how long a payment may be held is asserted anywhere in this kit, because none was researched.
• It is not a check on the model. It asks a question about the payment; whether the reply is self-consistent is a separate diagnostic and is reported separately.

WatchedWhat is watched, and why that one

The kit measures 25 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 18 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (failures 0, refusal_cells 0, thinking True) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

extraction · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on documents, extraction_cells, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

comply · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 3 chips that all say so.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/extract.py, to the ordered checks in src/policy.py, or to data/policy.json's windows — all three move which rows come back current, which is the only value the condition reads. Re-run tools/build_corpus.py to reprint the candidate table whenever the corpus changes.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt names nothing, and the file says why: a requirements line the code never imports costs a forker an install and teaches them the kit is bigger than it is.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

segment -> select (union of mapped sections, minus NEVER_SENT) -> prompt (system + policy + schema + file) -> one completion -> parse -> the currency join in src/policy.py -> the payment-level guardrail in src/extract.py -> five graders in evals/judge.py. Everything before the completion and everything after it is pure code.

The other sideWhat a framework costs you

• Writing the adapter by hand costs about 220 lines and buys provider independence plus a token count from every path, which lens 05 publishes and lens 07 prices.
• Having no rules engine costs an ordered if-chain in decide_item(); it buys a rule a forker can read in one screen and a single copy of it across the generator, the prompt and the scorer.
• Having no test framework costs evals/check_labels.py being a script rather than a suite; it buys a gate that runs with no key, no network and no install, before any run may spend.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-hold-reason on the fast tier, 2026-08-23. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 3 runs on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT, this repository's own licence, covering the corpus as well as the code Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-field exact match, light normalisation

Does the run's value for each field match gold, after trimming whitespace and punctuation? Four fields are legitimately null — last_verified, evidence_received, released_on and release_decided_by — so a null there is a hit when gold agrees. hold_cited_refs is compared as an ORDERED LIST rather than a string, because separator style is not the answer and which references are named is. THE DENOMINATOR IS GOLD'S ITEM BLOCKS: a block the run never returned costs its five cells rather than shrinking the denominator, because downstream a dropped block turns a cited reference into no_record_on_file.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every run, and the same field-match logic the free floor is scored by — a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

This row is the kit in one line. The flag is 13 days old against a 180-day window, so every clock on the page says it is fine — and it was set 6 days before the document it was waiting for was logged, so it is a reading of a folder that no longer exists. The payment is still stopped.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same values the file states; evals/check_labels.py asserts every non-nullable field is populated, that item references are unique within a file, that every gold cell can be re-read off its own file, and that the three payment states are internally consistent — a never-held payment carries no hold field, a released one names a date and a decider — before any run may spend.

How tight can the band be? There is no tolerance band — exact match after trimming whitespace and punctuation, never a continuous score to round.

Cadence: Re-score on any change to data/policy.json, src/prompt.py, data/fields.json or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe currency verdict, six-way, with the false-alarm rate

For every reference the hold notice names, did the run land on the same one of six currency values gold does — current, stale_elapsed, superseded_by_evidence, never_verified, no_record_on_file, or not_a_hold_reason? Read OFF the run's own worklist rather than re-derived, because re-running the join here would grade the policy instead of the pipeline. Its false-alarm rate and its missed actions are counted separately and never averaged together.

MethodHow the test was run

evals/judge.py::score_verdicts, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

This row is the kit in one line. The flag is 13 days old against a 180-day window, so every clock on the page says it is fine — and it was set 6 days before the document it was waiting for was logged, so it is a reading of a folder that no longer exists. The payment is still stopped.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/policy.py::decide_application() over gold's own items — the same function src/extract.py calls on the model's answer. evals/check_labels.py re-derives every gold row from it and fails if any disagrees.

How tight can the band be? Exact enum match. There is no partial credit for a near-miss value.

Cadence: Re-score on any change to src/policy.py's ordered checks or to data/policy.json's refresh windows.

The decisionWhen to reach for it

In one lineStale-flag recall, and its hardest third

Of the cited items gold says were NOT current when the hold was set, how many did the run report as not current? On its own denominator, never averaged into the six-way — a run can score well there and still miss the rows that keep a payment blocked. Its precision is counted too: a phantom stale is a hold declared unfounded when it was founded. And its hardest third, superseded_by_evidence, is scored separately on its own denominator, because that is the set the free floor is structurally 0 on.

MethodHow the test was run

evals/judge.py::score_verdicts, the stale_flags and superseded blocks.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

This row is the kit in one line. The flag is 13 days old against a 180-day window, so every clock on the page says it is fine — and it was set 6 days before the document it was waiting for was logged, so it is a reading of a folder that no longer exists. The payment is still stopped.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: gold's currency values from src/policy.py::decide_item(). The stale set is the three values data/policy.json names as stale; no_record_on_file and not_a_hold_reason are deliberately NOT in it and are reported as their own sets.

How tight can the band be? Recall and precision are reported separately and never combined into an F-score — the two errors have opposite consequences here, and one number hides which way a run is failing.

Cadence: Re-score on every run, and re-generate the corpus on any change to data/policy.json's windows, since they decide which rows are stale at all.

The decisionWhen to reach for it

In one lineThe hold verdict per payment, and the escalation

Per PAYMENT rather than per row: is at least one reason behind this hold still current? It is not an average of the rows — one current item is enough — so a run can be right about most rows and wrong about most holds. Alongside it, the payment-level escalation: still held, at least one item named, and not one of them current. Recall and precision named separately, plus a no-gold consistency diagnostic that re-runs the flag over the run's OWN rows.

MethodHow the test was run

evals/judge.py::score_holds and ::score_flag, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

This row is the kit in one line. The flag is 13 days old against a 180-day window, so every clock on the page says it is fine — and it was set 6 days before the document it was waiting for was logged, so it is a reading of a folder that no longer exists. The payment is still stopped.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: gold's well_founded from decide_application() and gold's escalate from src/extract.py::compute() — the same function the run uses, so gold and the run cannot drift about what the condition means. check_labels.py asserts they agree.

How tight can the band be? Boolean against boolean. An unanswered hold counts AGAINST the run rather than shrinking the denominator — which is what stops the free floor, which answers none of the 40, from scoring perfectly on nothing.

Cadence: Re-score on any change to compute() in src/extract.py, and re-run the candidate table in tools/build_corpus.py whenever the corpus changes.

The decisionWhen to reach for it