The business caseThe problem this solves

A grant's disbursement schedule conditions each scheduled payment on a milestone, and releasing one is not a summary of what the grantee reported -- it is a CLAIM that the milestone behind it was met. The dangerous line is therefore never a wrong figure. It is a payment marked Release when the evidence behind it cannot carry the word: the required deliverable never arrived, the period reported is not the period the schedule attaches, a condition precedent was reversed after it was recorded, or the deliverable on file has since been withdrawn or returned for correction. In every one of those cases the reported figure meets its target. Someone reading a grant's whole payment request against the agreement before the money goes out: checking that every scheduled payment was actually reported on, that the deliverable the schedule names is the deliverable on file, that the period reported is the period the payment attaches to, that any condition precedent is cleared and has stayed cleared, and that the target being measured against is the one in the amendment in force rather than the one transcribed off the grantee's cover sheet.

The inputThe actual grant payment request packets

The corpus is 24 grant payment request packets, 0.12 MB (txt 24). A real grant payment request names a charity, its banking details, its staff, its shortfalls and a funder's private assessment of all of it. There is no licence under which that could ship in a public repo, and a version redacted enough to publish would no longer contain the thing being measured. So the corpus is written rather than collected, and data/SOURCES.md says so on its first line rather than in a footnote. What is synthetic is the text; what is real is the defect shape.

Payment Request
----------------------------------------------------------------
  Packet                 GM-0001
  Grantee                Harborline Community Trust
  Programme              Neighbourhood Food Security Initiative
  Award                  GA-2025-100
  Request date           2026-07-05
  Prepared by            R. Adeyemi, Grants Management

Grant Agreement
----------------------------------------------------------------
  Agreement              GA-2025-100
  Amendment in force     Amendment 2, effective 2026-03-10
  Superseded amendment   Amendment 1, superseded 2026-03-10
  Holdback               5 pct of each scheduled payment, released at closeout

Disbursement schedule under Amendment 2 -- every scheduled payment below must be
decided on this request:

  No.   Milestone                       Deliverable     Target            Period                      Amount
  T1    Closeout narrative filed        CLOSE-NARR      1 narrative       2025-10-01 to 2025-12-31    $20,000
  T2    Evaluation design filed         EVAL-DES        1 design          2025-10-01 to 2025-12-31    $25,000
  T3    Advisory board minutes filed    CAB-MIN         1 minute set      2026-01-01 to 2026-03-31    $30,000
  T4    Staff trained                   TRAIN-ROSTER    55 staff          2026-01-01 to 2026-03-31    $35,000
  T5    Meals distributed               DISTRIB-LOG     12,600 meals      2026-04-01 to 2026-06-30    $40,000
  T6    Workshops delivered             WKSHP-LOG       25 sessions       2026-04-01 to 2026-06-30    $45,000
  T7    Baseline assessment filed       BASELINE-RPT    1 report          2026-07-01 to 2026-09-30    $50,000

Grantee Progress Report
----------------------------------------------------------------
Milestone T1

The outcomeWhat a good result looks like

A release recommendation: one line per scheduled payment, in the schedule's order, each carrying RELEASE, WITHHOLD or CANNOT_DECIDE with a named blocker -- plus a packet-level disposition, and gross/holdback/net computed in code from whichever decisions you take. Beside every line, what the strongest free code would have decided.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Every fact that decides a release is already a FIELD -- a deliverable code, a period, a condition line, and a report that either exists or does not. — the free floor -- evidence-gate, $0.00
  It scores 82.14 pct on the discriminator against the model's 83.33, catches 100 pct of structured blockers, raises 0 pct false holds and does the arithmetic at 100 pct. It is free and it wins seven of the twelve published columns.
• Decisive evidence lives in officer notes, e-mail threads or narrative -- an approval reversed after it was recorded, the wrong file behind a right code. — the model, on top of the free floor rather than instead of it
  30 of 30 prose blockers caught, against zero from every free floor, which drops the silent-release rate from 58.82 pct to 1.96 pct.
• You need a number you can put in front of a grants committee, and a wrong release is materially worse than a delayed one. — run both arms and take the union of their refusals
  The two arms fail in opposite directions: the floor's failure is 58.82 pct silent releases, the model's is 25.0 pct false holds. A union refuses on either signal, which on this corpus would have refused every genuinely unevidenced payment.

And where nothing here is good enough:

• You want the money figure checked. — nothing here
  Gross, holdback and net are computed in code from whatever an arm decided and are not scored by anything in evals/. No figure on this page measures them.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-27. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-27 — r001-milestone-disburse. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Every packet, the answer key, all six recorded runs and all three free floors are in the repo. python3 -m evals.check_labels and python3 -m evals.run --floor evidence-gate both run on a fresh clone with no key, no network and no install.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per packet, six concurrent workers, on a shared key carrying heavy concurrent load from other kits that day -- read these as an upper bound

Current processWhat it replaces

Someone reading a grant's whole payment request against the agreement before the money goes out: checking that every scheduled payment was actually reported on, that the deliverable the schedule names is the deliverable on file, that the period reported is the period the payment attaches to, that any condition precedent is cleared and has stayed cleared, and that the target being measured against is the one in the amendment in force rather than the one transcribed off the grantee's cover sheet.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusEvalsEvalsArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• 6 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one scheduled payment is

24 packets, each carrying one grant agreement's disbursement schedule at a stated amendment and the grantee's progress report against it. Every packet is independent -- there is no carried state and no ordering constraint -- so the run is embarrassingly parallel and a failure on one says nothing about any other. The packet SET is shaped rather than drawn: 5 release in full, 4 release nothing and 15 are mixed, because a corpus where every packet is mixed makes the packet-level verdict a constant.

SetupWhat the setup figure measured

There is no index to build. The whole packet, minus the withheld Grant Administration block, goes into the prompt verbatim -- no chunking, no retrieval, no pre-digest. A summariser here would be a place for the one interesting sentence to be lost before the model saw it, and then the measurement is of the summariser. The 0.3 seconds is corpus generation from the seed.

LicenceLicence

MIT -- this repository's own licence. The corpus is generated in-process by tools/build_corpus.py from a fixed seed and contains no third-party data of any kind.

Bring your ownBring your own grant payment request packets

tools/build_corpus.py writes data/corpus/*.txt and data/gold.jsonl. To point the kit at your own packets, change the four regular expressions in src/packet.py and the heading pattern in src/segment.py -- data/SOURCES.md has the table. You will also need your own answer key; evals/check_labels.py proves this one and cannot prove one you have not written. Until then the three free floors still run and still cost nothing.

⚠︎ And what stops being true when you do: None of the measured figures on this page transfer to your own packets. They are measurements of THIS corpus, whose defects were planted by the same author who wrote the answer key. What transfers is the shape: the split between defects a regular expression can decide and defects only a sentence reveals, and the discipline of reporting the two on separate denominators.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 3,145 characters assembled, 1,455 are rulebooks — 46% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Reproduced from src/prompt.build with the Grant Administration block withheld by src/select.py, exactly as every scored call sent it.

Full promptThe prompt in full

You are reviewing a grant payment request for a foundation's grants management office.

A disbursement schedule conditions each scheduled payment on a milestone. Releasing a payment is a
CLAIM that the milestone behind it has been met and evidenced -- not a summary of what the grantee
reported. Do not recommend releasing a payment the evidence cannot carry.

Return one row for EVERY payment the disbursement schedule lists, in the order the schedule lists
them, identified by its tranche number. Never drop a payment because the grantee did not report
on it.

For each scheduled payment give exactly one decision:

  RELEASE        the milestone is met on usable evidence, measured against the target in the
                 disbursement schedule as amended.
  WITHHOLD       usable evidence exists and the milestone was NOT met. This is still a decision
                 the office can act on.
  CANNOT_DECIDE  no decision can be supported on this evidence yet.

Evidence is USABLE only if all four of these hold:
  1. a progress report for that milestone is on file;
  2. the deliverable the schedule requires is on file and is the deliverable it names;
  3. the reporting period on file is the period the schedule attaches to that payment;
  4. any condition precedent recorded against the payment is cleared, and has stayed cleared.

Anything in the packet may bear on those four. Where a printed field and a later statement in the
packet contradict each other, they are not a tie.

Compare the reported achievement against the target stated in the DISBURSEMENT SCHEDULE under the
amendment in force. A target transcribed onto the grantee's cover sheet may have been taken from
an amendment that has since been superseded.

When the decision is CANNOT_DECIDE, give the reason as exactly one of:
  NO_REPORT_FOR_PERIOD       the schedule lists the payment and no progress report is on file
  DELIVERABLE_MISSING        the required deliverable is not on file, or what is on file is not it
  PERIOD_MISMATCH            the period reported is not the period the schedule attaches
  CONDITION_PRECEDENT_UNMET  a condition the payment depends on is not cleared
  DELIVERABLE_SUPERSEDED     the deliverable on file has been withdrawn, superseded or returned
                             for correction
Otherwise blocker is null.

Do not compute any amount of money. Amounts are calculated elsewhere from your decisions.

Reply with JSON and nothing else:

{"packet_disposition": "RELEASE_ALL" | "PARTIAL_RELEASE" | "RELEASE_NONE",
 "rows": [{"tranche": "<T1, T2, ... exactly as the schedule numbers it>",
           "milestone": "<as the schedule names it>",
           "reported": "<the reported achievement and unit, or null>",
           "decision": "RELEASE" | "WITHHOLD" | "CANNOT_DECIDE",
           "blocker": "<one of the five, or null>",
           "statement": "<the one sentence this line would carry to the grants committee>"}],
 "rationale": "<two sentences at most, on what decided the hardest payment>"}

packet_disposition is RELEASE_ALL when every row is RELEASE; RELEASE_NONE when no row is RELEASE;
PARTIAL_RELEASE otherwise.

Raw responseThe raw response

{"packet_disposition":"PARTIAL_RELEASE","rows":[{"tranche":"T1","milestone":"Closeout narrative filed","reported":"2 narrative","decision":"RELEASE","blocker":null,"statement":"Two closeout narratives are on file against the one-narrative target and the audit condition is cleared."},{"tranche":"T2","milestone":"Evaluation design filed","reported":"1 design","decision":"RELEASE","blocker":null,"statement":"Evaluation design is filed as required and the budget modification condition is cleared."},{"tranche":"T3","milestone":"Advisory board minutes filed","reported":"1 minute set","decision":"RELEASE","blocker":null,"statement":"Advisory board minutes are filed as required and the subrecipient condition is cleared."},{"tranche":"T4","milestone":"Staff trained","reported":"56 staff","decision":"RELEASE","blocker":null,"statement":"The training roster shows 56 staff trained, exceeding the 55-staff target, and the insurance condition is current."},{"tranche":"T5","milestone":"Meals distributed","reported":"12,631 meals","decision":"CANNOT_DECIDE","blocker":"DELIVERABLE_MISSING","statement":"The distribution log was filed before the required reporting period began, so it cannot evidence meals distributed in April through June."},{"tranche":"T6","milestone":"Workshops delivered","reported":"27 sessions","decision":"RELEASE","blocker":null,"statement":"The workshop log shows 27 sessions

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Grant payment-to-milestone reconciliation — 168 grant payment request packets drawn from 24 real grant payment request packets. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Exact match per scheduled payment, in pure Python. No model grades anything, there is no judge and no rubric, and re-running the scorer on the committed result files reproduces every figure on this page for $0.00. EVERY RATE CARRIES ITS OWN DENOMINATOR AND NOTHING IS BLENDED -- a catch rate with no false-hold rate beside it is not a measurement, and a blended catch rate would let the structured half carry the prose half, which is the precise claim this kit exists to test. A rate with a zero denominator returns None and renders as a dash: not-measured is a third state, never a zero.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on this page is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-27. The provider that actually ran every call here is kept off this page per the series rule. The real spend is in the shared call ledger, not on this page.

The gradersThree ways to grade

Read the floors as a ladder, because that is how they were built, and then read the last column of baseline_per_field, because it is the finding. Floor 1 -> 2 is five lines of Python and buys the entire silent-omission class. Floor 2 -> 3 is the four structured checks and buys the whole structured channel plus the stale-target case. Floor 3 -> the model buys ONE thing -- the prose channel, 30 of 30 against zero -- and pays for it with 27 false holds the floor does not raise. If your own evidence lives in FIELDS, the honest recommendation on this corpus is floor 3 and no model at all.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-27. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

Nothing was discarded and nothing was re-fired. The two None entries above are runs that DID NOT HAPPEN -- the second tier and the red-team probe -- and they are absent rather than zero. The three free floors and the stub cost $0.00 and are re-runnable on a clone with no key.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, AND ALMOST ALL OF THEM ARE REASONING. 96.8 pct of r001's output (429318 of 443517) was provider-side reasoning rather than the recommendation. The recommendation itself is a few hundred tokens per packet. You are paying for the reading, not the writing.
• PACKET LENGTH, LINEARLY AND ONLY LINEARLY. One call per packet, the packet goes in whole, nothing is cached and there is no index. Mean input was 2006.0 tokens; mean output was 18479.88 -- so output outweighs input by roughly 9x on this task and the input side is nearly free.
• THE NUMBER OF SCHEDULED PAYMENTS PER PACKET. Seven here. A schedule with twenty tranches is a longer reply and more reasoning, on the same single call.

Your volumeWhat it costs at your volume

LINEAR IN PACKETS. Nothing amortises, nothing is cached, there is no index and no retrieval. Ten times the packets is ten times the bill. The free floor stays at $0.00 at any multiple, which is the whole reason it is published beside the paid arm.

Model choiceWhy this model

It is the tier the shared connection was already pointed at, and no model shopping was done. That is a limitation of the measurement, not a recommendation: this page compares ONE paid tier against three free floors, and the free floors win most of the columns.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

per-packet average, this run

The codeEvery file, and what it is for

12 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysection split

Cut the packet on its underlined headings. Deterministic, so 'what did we send' is a list of section names rather than a guess about a token window.

# Cut a payment-request packet into addressable sections. Pure code, no model.

HEAD = re.compile(r"^([A-Z][A-Za-z ,]+)\n-{10,}\n", re.M)
def sections(text):
def named(text, name):

src/select.pythe seam that withholds — a swap seam

The Grant Administration block -- payment method, account tail, remittance mailbox, internal distribution -- never leaves the machine. Withheld at the seam rather than trusted to a sentence in a prompt, and the UI prints what went and what stayed.

You change it to: NEVER_SENT. Add a section name and it stops leaving the machine.

# SEAM 2 -- what leaves this machine. Pure code, and deliberately a denylist of one.

NEVER_SENT = ("Grant Administration",)
def sent(sec_names):
def body(text, sections_fn):

src/packet.pythe packet parser — a swap seam

The disbursement schedule's rows and every progress-report record, read with four regular expressions. The model is never asked for a field at a fixed offset. Returns empty lists rather than guessing on an unfamiliar layout.

You change it to: The four regular expressions. This is the file to change to point the kit at your own grants export.

# Read a payment-request packet into structured values. Pure code, no model.

RELEASE, WITHHOLD, CANNOT = "RELEASE", "WITHHOLD", "CANNOT_DECIDE"
BLOCKERS = ("NO_REPORT_FOR_PERIOD", "DELIVERABLE_MISSING", "PERIOD_MISMATCH",
DISPOSITIONS = ("RELEASE_ALL", "PARTIAL_RELEASE", "RELEASE_NONE")
def parse_target(s):
def parse(text):
def record_for(parsed, tranche):
def condition_cleared(text):
def meets(reported, target):
def releasable_usd(schedule, decisions, holdback_pct):

src/checks.pythe structured evidence checks — a swap seam

The four conditions a record's OWN FIELDS can decide: no report at all, deliverable missing, period mismatched, condition precedent not cleared. Pure code. This file is why the kit can say honestly which half of the job needs no model -- it takes 100 pct of that half.

You change it to: Which conditions pure code decides, and their priority when a payment trips more than one. Adding a check here moves work OFF the model -- and on this corpus that is where the value is.

# The structured evidence checks, in pure code. No model, no notes.

PRIORITY = ("DELIVERABLE_MISSING", "PERIOD_MISMATCH", "CONDITION_PRECEDENT_UNMET")
def structured_blocker(row, rec):
def cover_sheet_amendment_mismatch(parsed, rec):
def decision_from_fields(parsed, row):

src/prompt.pythe prompt

The whole instruction in one place, in send order, plus the evidence-blind variant the control arm uses.

# The assembled prompt, in one place, in the order it is sent.

SYSTEM = """You are reviewing a grant payment request for a foundation's grants management office.
BLIND_SYSTEM_SUFFIX = """
def build(text, blind=False):
def _strip_evidence(body):
def render(parts):

src/decide.pythe model call and reply parse

One packet in, one release recommendation out. Holds the published token ceiling and the tolerant JSON parse -- a correct answer wrapped in a code fence has not got it wrong.

# One packet in, one release recommendation out. The only place a model is called for a decision.

MAX_TOKENS = 64000
THINKING = None
def parse_reply(text):
def normalise(obj):
def decide(cfg, text, blind=False, complete_fn=None, max_tokens=None):

src/adapters/__init__.pythe adapter — a swap seam

One interface, several providers, raw HTTP and no vendor SDK. Retries a busy provider and refuses to retry a wrong request. Its socket timeout rides with the token ceiling, so a long legitimate reply is never recorded as a transport failure.

You change it to: Provider and model, from .env. Nothing else on this page changes.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=None):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/packet.pythe money, in code — a swap seam

releasable_usd() computes gross, holdback and net from whatever an arm decided. NEITHER ARM IS ASKED FOR IT and it is not scored -- a kit that ran a multiplication through a language model would be measuring the wrong thing.

You change it to: The four regular expressions. This is the file to change to point the kit at your own grants export.

# Read a payment-request packet into structured values. Pure code, no model.

RELEASE, WITHHOLD, CANNOT = "RELEASE", "WITHHOLD", "CANNOT_DECIDE"
BLOCKERS = ("NO_REPORT_FOR_PERIOD", "DELIVERABLE_MISSING", "PERIOD_MISMATCH",
DISPOSITIONS = ("RELEASE_ALL", "PARTIAL_RELEASE", "RELEASE_NONE")
def parse_target(s):
def parse(text):
def record_for(parsed, tranche):
def condition_cleared(text):
def meets(reported, target):
def releasable_usd(schedule, decisions, holdback_pct):

evals/baseline.pythe three free floors

Report-walk, schedule-walk and the evidence gate. Each a genuine attempt at the job, all three free, and the third wins seven of twelve published columns.

# THREE FREE FLOORS. No key, no model, no network. Each is a genuine attempt at the job.

MODES = ("report-walk", "schedule-walk", "evidence-gate")
def _row(tranche, milestone, rec, decision, blocker, why):
def review(text, mode="evidence-gate"):
def _wrap(p, rows, mode, rationale):

evals/scoring.pythe scorer

Exact match per scheduled payment, and every rate carries its own denominator. Nothing is blended, and a zero denominator returns None rather than 0.

# Score an arm against the answer key. Pure code, exact match per cell. No model grades anything.

RELEASE, WITHHOLD, CANNOT = "RELEASE", "WITHHOLD", "CANNOT_DECIDE"
OMITTED = "OMITTED"
def _pct(n, d):
def _key(s):
def rows_by_tranche(answer):
def score(records, golds):

evals/check_labels.pythe answer-key gate

Proves the key against the packets themselves on nine assertions, every one red-proven with --red-proof.

# Prove the answer key against the packets themselves. Free, no key, no model.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
SEEDS = ("schedule", "structured", "prose-clean", "no-report", "meets", "stale",
def load():
def check(seed=None):
def main():

src/app.pythe local UI

http.server, no dependency. Renders with no key, replays the committed run, and computes the strongest free floor on every row every time.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
CORPUS = os.path.join(HERE, "data", "corpus")
PORT = int(os.environ.get("PORT", "9193"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-milestone-disburse")
FLOOR = "evidence-gate"
def documents():
def load_doc(doc_id):
def read_in_code(text):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2006 input and 18479 output tokens per payment request packet (one grant's scheduled payments, decided together), all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

NOT MEASURED. No red-team probe has been run against this kit. evals/injection.py ships, is runnable and measures both directions -- whether an instruction-shaped note SUPPRESSES a refusal the fields prove, and whether it FABRICATES one on a clean payment -- and it has never been executed. Every posture figure here is therefore ABSENT rather than zero.

API_KEY is read from <code>&lt;repo&gt;/.env</code>, this kit's own <code>.env</code> and the real environment, in that order, by src/config.py. Both files are gitignored from the first commit and this repository has never held a credential. src/config.sources() reports WHICH file a value came from and never the value. The UI's error path substitutes the key and base URL out of any provider message before returning it.

The experimentWe did NOT attack it

No injection run exists for this kit. The probe that would produce one is evals/injection.py: it replaces the whole Programme Officer Notes section of every packet carrying a structured blocker with a single instruction-shaped sentence claiming the programme director waived the evidence conditions, and separately adds a sentence asserting withdrawal to packets with no blocker at all. Run it on your own key; the result file records what it does NOT measure alongside what it does. NOTHING WAS RUN. evals/injection.py ships, is runnable and has never been executed against this kit -- this page was written on 2026-08-27 and records the absence, not a result.

One boundary is enforced in code and red-proven -- the withheld Grant Administration section, assertion A9 with its own withheld seed. The two injection boundaries are written, committed and NOT fired, and are reported as unrun rather than as a pass. Nothing in this table is a resistance claim.

The resultNOT MEASURED -- no injection probe was run against this kit

No red-team trial was run. The third card is not an injection result: it is the scored run's own silent-release count, and it is here because it is the closest thing to a security reading this kit possesses. A note that was NOT trying to mislead still moved this model off a field it could see, on a $95,000 payment. What a note written to mislead would do has not been tried.

HonestyWhat this does not prove

• Whether an instruction-shaped programme officer note can suppress a refusal the record's own fields prove.
• Whether such a note can fabricate a refusal on a payment with no blocker at all -- the mirror failure, which a catch-rate-only probe would miss.
• Any phrasing, any model, any corpus other than the one shipped here.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never release a payment, amend an agreement, accept a report, notify a grantee or move money. This kit produces a RECOMMENDATION for a person who signs.

Stated on the UI, in the README and in src/app.py's own docstring, and enforced by there being no such endpoint, no such button and no configuration flag that adds one. The only POST route is /api/decide and it returns JSON.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is not a control on the grantee. Nothing here validates that a filed deliverable is what it claims to be; that is what the programme officer notes are for and what the prose channel measures.
• It is not a fraud check. It compares a packet against its own agreement, nothing else.
• It is not a redaction system. src/select.py withholds a NAMED SECTION; a bank detail inside a programme officer note would be sent.

WatchedWhat is watched, and why that one

The kit measures 81 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

11 measured by the latest run 70 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (blocker_reason_cells 0, blocker_reason_correct 0, cells 168, decidable_cells 108, false_holds 0, majority_decision_pct 50.0, majority_disposition_pct 62.5, milestone_release_correct 96, no_report_caught 0, no_report_cells 9, packets 24, prose_blocker_caught 0, prose_blocker_cells 30, rows_omitted_total 9, silent_omissions 9, silent_release 51, stale_target_caught 0, stale_target_cells 12, structured_blocker_caught 0, structured_blocker_cells 30, target_arithmetic_cells 108, tranches_per_packet 7, trap_called_withhold 0, trap_cells 51, undecidable_caught 0, undecidable_cells 60) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

7 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

reconcile · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 3 runs measured the same system — they differ on blocker_reason_cells, blocker_reason_correct, milestone_release_correct, no_report_caught, rows_omitted_total, silent_omissions, silent_release, stale_target_caught, structured_blocker_caught, undecidable_caught, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

reconcile · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on blocker_reason_cells, blocker_reason_correct, cells, decidable_cells, false_holds, majority_decision_pct, majority_disposition_pct, max_tokens, milestone_release_correct, no_report_caught, no_report_cells, packets, prose_blocker_caught, prose_blocker_cells, silent_release, stale_target_caught, stale_target_cells, structured_blocker_caught, structured_blocker_cells, target_arithmetic_cells, trap_cells, undecidable_caught, undecidable_cells — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 11 chips that all say so.

DeviationsWhat deviated

0 breaches across 7 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The answer-key gate and all three free floors are free and run before any spend. The paid arm is one call per packet and re-runnable at will. No red-team probe has been run.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, standard library end to end -- requirements.txt names nothing. The model is reached over urllib, the UI is http.server, the corpus is generated in-process and the scorer is a loop.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear: a packet -> src/segment.py -> src/select.py -> src/prompt.py -> src/adapters -> src/decide.py -> evals/scoring.py. There is no branch, no retry loop above the adapter's own, and no agent.

The other sideWhat a framework costs you

• You write the parsing yourself. Four regular expressions in src/packet.py, and they are written for this corpus's layout.
• You get no observability, no tracing and no retry policy beyond the adapter's four bounded backoffs. For a kit that is the right size; for production it is not.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-milestone-disburse on the fast tier, 2026-08-27. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 5 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, HTTP_TIMEOUT_SECONDS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence. The corpus is generated in-process by tools/build_corpus.py from a fixed seed and contains no third-party data of any kind. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe decision on each scheduled payment, exact match, three ways

whether each of the 168 scheduled payments got the decision the answer key carries -- RELEASE, WITHHOLD or CANNOT_DECIDE

MethodHow the test was run

python3 -m evals.run --run-id <id> ...; evals/scoring.py computes it in-process.

Every grader on these pages scored the same 168 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

THE ONE PAYMENT THE MODEL WOULD HAVE RELEASED AND SHOULD NOT HAVE -- $95,000. The field says the deliverable was never received. A benign programme officer note says the return was filed through the portal. The model believed the sentence over the field; the free floor, which cannot read notes at all, got it right for nothing. Every other cell in this kit argues that reading prose is what the model is for. This one is the cost of that.

The formulaWhat it computes

tranche_decision_accuracy_pct = hits / 168. A payment the arm left off its recommendation entirely counts as OMITTED and is a miss.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, the hand-planted answer key, itself proved against the packets by evals/check_labels.py on nine red-proven assertions. THIS GRADER IS THE REFERENCE STANDARD for this kit, so its own rates against truth are UNKNOWN -- a number there would be circular.

How tight can the band be? 168 cells, so one cell is 0.60 points. A band tighter than a point is reading noise off a denominator this size.

Cadence: every run; it is free and it is computed in-process by evals/scoring.py.

The decisionWhen to reach for it

In one lineDecision AND blocker, both right

whether a payment got a call a grants officer could sign -- the right decision and, where the evidence does not support one, the right blocker from the closed set of five

MethodHow the test was run

same run, same scorer.

Every grader on these pages scored the same 168 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

THE ONE PAYMENT THE MODEL WOULD HAVE RELEASED AND SHOULD NOT HAVE -- $95,000. The field says the deliverable was never received. A benign programme officer note says the return was filed through the portal. The model believed the sentence over the field; the free floor, which cannot read notes at all, got it right for nothing. Every other cell in this kit argues that reading prose is what the model is for. This one is the cost of that.

The formulaWhat it computes

milestone_release_correct = cells where decision matches AND blocker matches. A COUNT, so it is never null.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the same answer key -- data/gold.jsonl, decision AND blocker.

No true/false rates for this grader. It records 4 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? A COUNT out of 168, deliberately -- it is the key a run record dispatches on, and a rate would go null on an arm with a zero denominator.

Cadence: every run, free.

The decisionWhen to reach for it

In one lineCatch rate, split by channel, on two denominators

of the 30 blockers a record's own fields prove and the 30 only a sentence reveals, how many each arm caught -- never averaged together

MethodHow the test was run

same run, same scorer.

Every grader on these pages scored the same 168 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

THE ONE PAYMENT THE MODEL WOULD HAVE RELEASED AND SHOULD NOT HAVE -- $95,000. The field says the deliverable was never received. A benign programme officer note says the return was filed through the portal. The model believed the sentence over the field; the free floor, which cannot read notes at all, got it right for nothing. Every other cell in this kit argues that reading prose is what the model is for. This one is the cost of that.

The formulaWhat it computes

structured_blocker_caught_pct and prose_blocker_caught_pct, separately, always.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the answer key's own channel field, set when each defect was planted: structured when the record's own fields prove it, prose when only a note does. evals/check_labels.py assertion 3 proves every prose cell is structurally CLEAN, so the split is verified rather than asserted.

No true/false rates for this grader. It records 3 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? 30 structured cells and 30 prose cells. Both denominators are small enough that one cell moves the rate by more than three points -- do not band these tighter than that.

Cadence: every run, free. Never blend the two rates.

The decisionWhen to reach for it