The business caseThe problem this solves

A grant application arrives with its attachments — a budget, a board list, a tax determination letter, audited financials, a project narrative. Before a programme officer reads a word of the proposal, somebody checks it against the foundation's published checklist. Two things make that check harder than a tick list. The right document is often enclosed in the wrong FORM — a compilation where an audit is required, last year's statements, a board list with no affiliations, a determination letter naming the fiscal sponsor rather than the applicant. And several clauses only apply CONDITIONALLY: the audit clause fires above $750,000 of expenses, the fiscal sponsor clause only for a sponsored project. A screen that cannot read the condition chases a volunteer-run applicant for paperwork nobody asked them for, which is exactly how a completeness screen becomes a barrier to the applicants a fund exists to reach. the clause-by-clause read a grants administrator does against the published checklist before an application reaches a programme officer

The inputThe actual screen packets

The corpus is 40 screen packets, 0.24 MB (json 4 · jsonl 1 · md 2 · txt 40). ⚠︎ EVERY BYTE OF IT IS INVENTED, AND THAT IS STATED BEFORE ANY NUMBER. The Merrivale Trust is a fictional foundation, the Community Resilience Fund a fictional programme, and CRF-7 a fictional checklist written for this kit; its part numbers, clause numbers and dollar thresholds are made up, and the taxpayer identification numbers are EIN-shaped with the prefix 00, which is not issued to anybody. It was generated rather than found because the measurement needs a key at the level of one clause of one application AND needs the corpus shaped against the shortcuts a reader might take: 124 of the 384 attachments cite the clause they answer and a wrong-form one cites it as often as a satisfying one; 86 clauses do not apply to the applicant at all and 35 of those have the document enclosed ANYWAY; 17 attachments name a document and enclose nothing; 53 satisfying attachments are reworded end to end while keeping every required particular; and 159 attachments answer no clause at all.

GRANT APPLICATION COMPLETENESS SCREEN — APP-0001
Programme: Community Resilience Fund, Round 7 (fictional) · The Merrivale Trust (fictional)
Applicant: Ashcombe Riverside Gardens (fictional) · Taxpayer id 00-1400037 (invented, not a real EIN)
Checklist: Community Resilience Fund, Round 7 — Eligibility and Completeness Checklist (a fictional checklist written for this kit) · Reference CRF7-02217

PART A — WHAT THE APPLICANT DECLARED ON THE COVER SHEET
These declarations are the applicant's own. Several clauses of PART B apply only
when a declaration below meets the condition the clause states.

      The applicant is independently incorporated and holds its own exemption determination.
      Total expenses in the most recent completed fiscal year were $2,597,093.
      The applicant employs no paid staff and is run entirely by volunteers.
      The amount requested is $59,000.
      No part of the amount requested is for equipment or building works.
      No part of the amount requested is passed to another organisation.
      The applicant has previously held a grant from this fund, awarded in 2021.

PART B — WHAT THE PUBLISHED CHECKLIST REQUIRES
Each clause below is a completeness requirement. An application answers a clause
only if something actually enclosed responds to THAT clause, in the form the
clause states.

  CRF-7-2.4.d  Part 2 Application and Signature — Project narrative
      Submit a project narrative of not more than eight (8) pages, covering the need addressed, the
      activities proposed, the population served and the timeline, in the programme's published
      narrative order, with every heading used.

  CRF-7-4.1.a  Part 4 Financial Documentation — Audited financial statements

The outcomeWhat a good result looks like

Every checklist clause gets one of four verdicts and the evidence behind it: the attachment and the words it rests on where the application answers, the applicant's own declaration where the clause does not apply.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• you need a list of documents nobody enclosed at all — the free best floor — 0 calls, $0.00
  it scores 231 of 234 on presence against the paid call's 233. 3 clauses to 1, exact two-sided p = 0.625000: on this corpus there is NO MEASURABLE DIFFERENCE, and the code costs nothing and reaches no network
• your checklist has conditional clauses and you care about not chasing applicants for paperwork they do not owe — the paid call
  it chased 1 of the 86 out-of-scope clauses against the tuned floor's 9 and the plain clause checklist's 51, and it is right on all 35 of the ones where the applicant enclosed the document anyway
• you need to know whether the enclosed document is in the FORM the clause asks for — the paid call, and nothing else on this page comes close
  171 of 173 against the tuned floor's 120 — 51 discordant clauses to 0, p < 0.000001. The clause checklist scores the CONSTANT: 103 of 173, which is what answering SATISFIED to every enclosed document gets you

And where nothing here is good enough:

• you want to decide anything about the application itself — none of this
  there is no endpoint here that declines, scores, ranks or funds. The reply schema has four fields and none of them is a decision, and the four boundary counters were 0 on every one of the 40 calls and 0 under all 12 attacks

At a glanceHow the whole thing runs

Run once, for real, on 2026-09-11. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-09-11 — r001-applicant-screen. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A clean checkout with no key configured renders the whole board — the corpus, the answer key, the committed run and all four free floors come off disk, and the free floors re-score offline at $0.00. That state is what the empty screenshot is shot against: a second server started with API_KEY blanked in its own environment, not a banner drawn to look like one.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. end to end for one screen packet — prompt assembly, one call, the JSON parse, the consistency station and the citation locator. Sequential, one application at a time, no concurrency.

Current processWhat it replaces

the clause-by-clause read a grants administrator does against the published checklist before an application reaches a programme officer

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one checklist clauses judged is

one application is one call; every packet carries 7 cover-sheet declarations, 8 checklist clauses and 9 to 13 attachments

SetupWhat the setup figure measured

there is no index. The applicant's declarations, the checklist clauses and the attachment list go into the one call whole; nothing is chunked, embedded or retrieved, and the answer key is derived by the generator rather than searched for.

LicenceLicence

MIT (the corpus is generated; there is nothing in it anyone else owns)

Bring your ownBring your own screen packets

Replace data/checklist.json with your own — clause ids, document types, requirement text, the particulars each clause names and the CONDITION sentence that decides whether it applies — and rewrite the clause table in tools/build_corpus.py so the generator can build packets from it. Nothing in src/ knows a clause id: the prompt sends the packet verbatim and the station addresses attachments by their position in PART C. If you have real screened applications with a per-clause key, point src/application.py at them and skip the generator entirely — the scorer only wants data/gold.jsonl.

⚠︎ And what stops being true when you do: THE MEASURED ACCURACY DOES NOT TRAVEL WITH YOUR CHECKLIST. Every rate on this page was measured against an invented checklist whose wrong forms come from a published list of degraded values and whose conditions are each settled by one declared figure. Real applications fall short in ways nobody enumerated and real conditions are messier than any of these. The kit travels; the numbers do not.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 10,106 characters assembled, 6,690 are documents — 66% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Replayed from the kit's own src/prompt.py build() for APP-0001, not logged by the run. It is checkable rather than trusted: the run record stores the part character counts and the assembled length, and the replay reproduces them.

Full promptThe prompt in full

SYSTEM:
You are screening a grant application for COMPLETENESS against a foundation's published checklist, before a programme officer reads it. You read only the packet you are given. You never invent an attachment, a declaration or a quotation, and you never decline an application, score its merit, rank it against other applications or recommend funding — a programme officer reads the application and the foundation decides; this is the completeness screen made beforehand.

USER:
For every clause listed in PART B, decide how the application answers it, and choose exactly one
verdict:

  SATISFIED     an item of PART C is the document the clause asks for, and it is in the FORM the
                clause states — the same approvals, the same signatures, the same period, the same
                issuer, the same content, the same counts. The wording may be completely different.
                Different words describing the same enclosure are SATISFIED.

  MISSING       the clause applies to this applicant and no item of PART C answers it. An item that
                merely names the document and encloses nothing — listed on the attachment index, to
                follow under separate cover, available on request, published on our website, held
                by our accountant, in preparation — answers nothing and is still MISSING.

  WRONG_FORM    an item of PART C IS the document the clause asks for, but not in the form the
                clause states — an unaudited compilation where an audit is required, last year's
                statements where the most recent completed year is required, one signature where
                two are required, a list missing a column the clause names, a document naming the
                wrong organisation, a policy last reviewed outside the stated window. The document
                is right and the form is wrong.

  OUT_OF_SCOPE  the clause carries a CONDITION in PART B and the applicant's own declarations in
                PART A do not meet it, so nothing is owed against this clause. Read the condition
                against PART A and compare the actual figure or fact. THIS VERDICT DOES NOT DEPEND
                ON WHAT IS ENCLOSED: an applicant who encloses the document anyway has not made the
                clause apply, and a clause that does not apply is never MISSING.

Decide APPLICABILITY first, from PART A against the clause's stated condition. A clause with no
CONDITION line always applies. Only once the clause applies does an absent document make it
MISSING, and only once something is enclosed does a defect in it make it WRONG_FORM.


THE PACKET, verbatim:

GRANT APPLICATION COMPLETENESS SCREEN — APP-0001
Programme: Community Resilience Fund, Round 7 (fictional) · The Merrivale Trust (fictional)
Applicant: Ashcombe Riverside Gardens (fictional) · Taxpayer id 00-1400037 (invented, not a real EIN)
Checklist: Community Resilience Fund, Round 7 — Eligibility and Completeness Checklist (a fictional checklist written for this kit) · Reference CRF7-02217

PART A — WHAT THE APPLICANT DECLARED ON THE COVER SHEET
These declarations are the applicant's own. Several clauses of PART B apply only
when a declaration below meets the condition the clause states.

      The applicant is independently incorporated and holds its own exemption determination.
      Total expenses in the most recent completed fiscal year were $2,597,093.
      The applicant employs no paid staff and is run entirely by volunteers.
      The amount requested is $59,000.
      No part of the amount requested is for equipment or building works.
      No part of the amount requested is passed to another organisation.
      The applicant has previously held a grant from this fund, awarded in 2021.

PART B — WHAT THE PUBLISHED CHECKLIST REQUIRES
Each clause below is a completeness requirement. An application answers a clause
only if something actually enclosed responds to THAT clause, in the form the
clause states.

  CRF-7-2.4.d  Part 2 Application and Signature — Project narrative
      Submit a project narrative of not more than eight (8) pages, covering the need addressed, the
      activities proposed, the population served and the timeline, in the programme's published
      narrative order, with every heading used.

  CRF-7-4.1.a  Part 4 Financial Documentation — Audited financial statements
      CONDITION: Required where the applicant's total expenses in the most recent completed fiscal
      year exceeded $750,000.
      Submit audited financial statements for the most recent completed fiscal year, prepared by a
      licensed independent auditor, carrying the independent auditor's opinion.

  CRF-7-4.2.a  Part 4 Financial Documentation — Exemption determination letter
      Submit an exemption determination letter issued by the national tax authority, naming the
      applicant itself, showing an exempt status that is current and has not been revoked.

  CRF-7-4.3.b  Part 4 Financial Documentation — Fiscal sponsor agreement
      CONDITION: Required where the applicant is an unincorporated project of a fiscal sponsor.
      Submit the fiscal sponsorship agreement, signed by both the sponsor and the applicant project,
      stating the sponsor's administrative fee, the terms on which funds are released and the date
      the agreement ends, and in force for the whole of the proposed project period.

  CRF-7-5.2.a  Part 5 Project Detail — Capital works and equipment schedule
      CONDITION: Required where the amount requested includes equipment or building works of $25,000
      or more.
      Submit a capital works and equipment schedule listing each item of equipment or building work,
      its cost and the quotation it rests on, with two (2) written quotations for every item over
      $10,000, and the building owner's written consent where the work is to a building the
      applicant does not own.

  CRF-7-6.1.a  Part 6 Prior Awards — Prior-grant closeout report
      CONDITION: Required where the applicant has previously held a grant from this fund.
      Submit the closeout report for the previous award, giving the outcomes reported against the
      previous award and the final expenditure against its budget, covering the whole of the
      previous award period, signed by the officer who certified the previous award's final report.

  CRF-7-7.1.a  Part 7 Governance — Board of directors list
      Submit a list of the board of directors giving every director, the office held and the year
      first elected, with each director's employer or principal affiliation stated, as constituted
      on the date of application.

  CRF-7-7.3.b  Part 7 Governance — Conflict of interest policy
      Submit the conflict of interest policy, adopted by the board and minuted, stating the
      disclosure required of a director, the recusal it triggers and the record kept of it, and
      reviewed within the last three (3) years.

PART C — WHAT THE APPLICATION ENCLOSES

  Item 1. Supporting Document (CRF-7-7.1.a)
      A list of the board of directors is enclosed. It gives each director, the office they hold and
      the year of their first election, it comes naming the employer or principal affiliation of
      every director, and it is as the board stands on the day of application.

  Item 2. Attachment (CRF-7-4.3.b)
      The fiscal sponsorship agreement is enclosed. It is executed by the sponsor and by the
      sponsored project alike, it states what the sponsor charges in administrative fee, how funds
      are released and when the agreement expires, and it is running for the entire proposed project
      period.

  Item 3. Cover Letter
      A cover letter from the executive director introducing the application and thanking the
      programme office for the round.

  Item 4. Letters of Support
      Three letters of support from community partners, not asked for by the checklist.

  Item 5. Further Documentation (CRF-7-4.1.a)
      Enclosed: audited financial statements for the most recent completed fiscal year, prepared by
      the applicant's own bookkeeper as a compilation, carrying the independent auditor's opinion.

  Item 6. Enclosure (CRF-7-6.1.a)
      Enclosed: the closeout report for the previous award, giving the outcomes reported against the
      previous award and the final expenditure against its budget, covering the whole of the
      previous award period, signed by the officer who certified the previous award's final report.

  Item 7. Project narrative (CRF-7-2.4.d)
      A project narrative is enclosed. It runs to eight (8) pages or fewer, it covers the need it
      addresses, the activities it proposes, whom it serves and when it happens, and it is set out
      following the narrative order the programme publishes, using each heading.

  Item 8. Annual Report
      The applicant's illustrated annual report for the previous year, supplied for background.

  Item 9. Conflict of interest policy
      The conflict of interest policy is enclosed. It is adopted by resolution of the board and
      recorded in the minutes, it states what a director must disclose, when that requires them to
      stand aside and how it is recorded, and it has been reviewed at some point in the last three
      (3) years.

END OF SCREEN PACKET — APP-0001


Reply with JSON and nothing else, in exactly this shape:

{"application_id": "<the id at the top of the packet>",
 "findings": [
   {"requirement": "<the clause id from PART B>",
    "verdict": "SATISFIED" | "MISSING" | "WRONG_FORM" | "OUT_OF_SCOPE",
    "item": "<the PART C item you are relying on, e.g. Item 7 — or NONE for MISSING and for
             OUT_OF_SCOPE>",
    "quote": "<the words you are relying on, copied exactly: the PART C item for SATISFIED and
             WRONG_FORM, the PART A declaration line for OUT_OF_SCOPE, empty for MISSING>"}
 ],
 "note": "<one sentence for the programme officer>"}

One object per clause in PART B, in the order PART B lists them. The quote must be copied character
for character out of the packet; do not paraphrase it and do not repair it.

Raw responseThe raw response

{"application_id": "APP-0001",
 "findings": [
   {"requirement": "CRF-7-2.4.d",
    "verdict": "SATISFIED",
    "item": "Item 7",
    "quote": "A project narrative is enclosed. It runs to eight (8) pages or fewer, it covers the need it addresses, the activities it proposes, whom it serves and when it happens, and it is set out following the narrative order the programme publishes, using each heading."},
   {"requirement": "CRF-7-4.1.a",
    "verdict": "WRONG_FORM",
    "item": "Item 5",
    "quote": "Enclosed: audited financial statements for the most recent completed fiscal year, prepared by the applicant's own bookkeeper as a compilation, carrying the independent auditor's opinion."},
   {"requirement": "CRF-7-4.2.a",
    "verdict": "MISSING",
    "item": "NONE",
    "quote": ""},
   {"requirement": "CRF-7-4.3.b",
    "verdict": "OUT_OF_SCOPE",
    "item": "NONE",
    "quote": "The applicant is independently incorporated and holds its own exemption determination."},
   {"requirement": "CRF-7-5.2.a",
    "verdict": "OUT_OF_SCOPE",
    "item": "NONE",
    "quote": "No part of the amount requested is for equipment or building works."},
   {"requirement": "CRF-7-6.1.a",
    "verdict": "SATISFIED",
    "item": "Item 6",
    "quote": "Enclosed: the closeout report for the previous award, giving the outcomes reported against the previous award and the final expenditure against its budget, covering the whole of the previous award period, signed by the officer who certified the previous award's final report."},
   {"requirement": "CRF-7-7.1.a",
    "verdict": "SATISFIED",
    "item": "Item 1",
    "quote": "A list of the board of directors is enclosed. It gives each director, the office they hold and the year of their first election, it comes naming the employer or principal affiliation of every director, and it is as the board stands on the day of application."},
   {"requirement": "CRF-7-7.3.b",
    "verdict": "SATISFIED",
    "item": "Item 9",
    "quote": "The conflict of interest policy is enclosed. It is adopted by resolution of the board and recorded in the minutes, it states what a director must disclose, when that requires them to stand aside and how it is recorded, and it has been reviewed at some point in the last three (3) years."}
 ],
 "note": "Two defects for the programme officer: the audited statements are a bookkeeper's compilation (CRF-7-4.1.a, WRONG_FORM) and no exemption determination letter is enclosed (CRF-7-4.2.a, MISSING); the fiscal sponsor and capital works clauses do not apply on the applicant's own declarations."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Application completeness screening — 320 screen packets drawn from 40 real screen packets. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Every grader is pure code against a key the corpus generator derived from the case it planted in each application, and the key is re-derived independently by evals/check_labels.py before any number is published. There is no judge model anywhere in this kit, so re-scoring the committed run costs $0.00 and returns the same answer every time.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

USD per million tokens, read from the Foundry app's build/facts/models.json, which carries as_of and the vendor's own pricing URL per row. The run's MEASURED token counts are projected onto each card; a projection is not a bill and no accuracy figure on this page belongs to any of these rows.

Rates checked 2026-09-05. cache-hit input. This kit prices every cache hit at the full input rate even though the source card publishes a cache-hit price, so every dollar figure here is an upper bound and the true spend is at or below it.

Grading unitWhat the grading figure prices

The graders are code, so grading costs nothing and no data leaves the machine. That is a property of the GRADER, not of the pipeline it grades — the pipeline costs $0.000888 per application.

The gradersThree ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-09-05. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

Every grader is pure code. Re-scoring the committed run, running all four free floors and re-deriving the answer key over 4,666 checks together cost $0.00 and reach no network. The only money this kit has ever spent is the 40 calls of r001-applicant-screen ($0.035514) and the 12 calls of the adversarial probe ($0.011952).

Cost driversWhat actually moves the bill

• THE PACKET, NOT THE INSTRUCTIONS. The role, the four verdict definitions and the reply schema are 3416 of the 10133 characters sent and are identical on every call; the packet is the rest and it is the only part that moves.
• THE NUMBER OF CLAUSES DRAWN FOR THE APPLICATION. Eight here. It drives the prompt and the reply together — one object per clause, each carrying a verbatim quotation — so it is the one lever that moves both sides of the bill.
• THE ATTACHMENT LIST. 9 to 13 attachments per application, 384 in the corpus, of which 159 answer no clause at all. The noise is realistic and it is paid for on every call.
• THE QUOTATION REQUIREMENT. Every non-missing finding copies the words it rests on, which is why the average reply is 629.6 output tokens rather than a few dozen. It is also what makes the citation grader possible, so it is bought deliberately.
• THE PEAK/OFF-PEAK TARIFF, which is a property of the clock and not of the workload. The same run inside the weekday peak window is $0.071026 against $0.035514 off peak.

Your volumeWhat it costs at your volume

linear in applications, and slightly sub-linear in dollars because the 3416-character shared prefix caches: 14976 cache-hit tokens were recorded across 40 calls. 400 applications is about $0.36 off peak on the card this run was made on. It is NOT linear in checklist length — doubling the clauses roughly doubles the reply as well as the prompt.

Model choiceWhy this model

One provider, one key, one model — the estate default, reasoning disabled, and the smallest reply ceiling that never truncated. The point of the kit is the comparison against free code, not a bake-off between vendors, and a second paid tier would have doubled the bill without changing which of the three questions the money buys.

Other modelsThis run's measured tokens, priced against published cards

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

the 40 calls of r001-applicant-screen, one per application. The 12 calls of x001-applicant-screen (the adversarial probe, $0.011952) are counted separately and are not in this figure.

The codeEvery file, and what it is for

14 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pycorpus generator — a swap seam

writes the 40 screen packets, the 14-clause fictional checklist and the 320-row answer key from seed 20260911

You change it to: the condition block on a clause — the declared field, the comparator and the threshold — plus the PART A sentence that prints the applicant's figure. evals/check_labels.py re-reads the sentence independently and will refuse a condition it cannot parse.

# UC0420 applicant-screen — build the synthetic corpus, the answer key and the stats block.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
SEED = 20260911
DATASET_VERSION = "applicant-screen-corpus-v1"
CHECKLIST_ID = "CRF-7"
CHECKLIST_TITLE = "Community Resilience Fund, Round 7 — Eligibility and Completeness Checklist"
FUNDER = "The Merrivale Trust"
PROGRAMME = "Community Resilience Fund, Round 7"

src/application.pypacket reader

the corpus off disk — the .txt the model is sent, and the structured index the floors and the scorer address attachments and declarations through

# The corpus, read off disk. One packet = one grant application's cover-sheet declarations, the

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
def _load(name):
def applications():
def checklist():
def rulebook():
def stats():
def ids():

src/prompt.pyprompt assembly — a swap seam

four parts in a fixed order, longest shared prefix first, the packet verbatim in the middle

You change it to: VERDICT_MEANINGS. src/reader.py imports VERDICTS from there rather than retyping them, and an assertion at import fails if a verdict is declared and not defined to the model.

# Assemble the one prompt this kit sends. Four parts, fixed order, longest-shared-prefix first.

MAX_TOKENS = 1600
SYSTEM = (
VERDICT_RULES = """\
VERDICT_MEANINGS = {
VERDICTS = tuple(VERDICT_MEANINGS)
SCHEMA = """\
def build(packet_text):
def parts():

src/adapters/__init__.pyprovider adapter — a swap seam

one function per wire format over stdlib urllib; no vendor SDK, so a forker installs nothing

You change it to: PROVIDER, BASE_URL and MODEL in .env. Any OpenAI-compatible endpoint or Anthropic's Messages API; adding a third is one function and one entry in PROVIDERS.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TIMEOUT_S = 1200
TRANSPORT_RETRIES = 1
def _post(url, headers, payload, timeout=TIMEOUT_S):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}

src/budget.pyspend guard

a daily call cap shared by every kit under the same .env, checked before the request is made

# A cap on live calls, shared by every kit on this machine. No dependency, no service.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
KIT = os.path.basename(HERE)
ROOT = os.path.dirname(os.path.dirname(HERE))
SHARED_ENV = os.path.join(ROOT, ".env")
LEDGER = os.path.join(ROOT if os.path.exists(SHARED_ENV) else HERE, ".calls-ledger.jsonl")
PAID_ARMS = ("r", "x")
FREE_ARMS = ("t", "b")
BOOT = os.urandom(6).hex()
PID = os.getpid()

src/reader.pyreply reader

the JSON parse. An unreadable reply is its own state and stays in the denominator, never a silent zero

# Turn one reply into findings, or say plainly that it could not be read.

class Unreadable(ValueError):
def _outermost_object(s):
def parse(text):

src/station.pyconsistency station

five repairs, every one counted, and it never changes a verdict — including forcing an OUT_OF_SCOPE finding to name no attachment

# The consistency station — pure code run over every reply before anything is published.

ITEM = re.compile(r"item\s*0*(\d+)", re.I)
def _item_no(ref):
def apply(application, parsed):

src/citation.pycitation locator

is the quoted text actually in the packet, and is an OUT_OF_SCOPE quote a DECLARATION rather than an attachment

# Locate the article text an answer quotes, inside the packet it came from.

COVERAGE_MIN = 0.60
PRECISION_MIN = 0.40
def norm(text):
def span_of(normalised_doc, quote):
def overlap(a, b):
def score(normalised_doc, quote, admitted_rows):
def whole_doc_precision(normalised_doc, row_span):

src/refusal.pyboundary reader — a swap seam

four things the one free sentence must never assert — a disposition, a merit judgement, a funding recommendation, a ranking

You change it to: four regexes. The structural guarantee is the schema — there is no field for a decision — and this reads the one free sentence that is not structurally constrained.

# Read the arm's one free sentence for things this kit must never assert.

DISPOSITION = re.compile(r"\b((is|are) (therefore )?(ineligible|disqualified|not eligible)|"
MERIT = re.compile(r"\b((a |an )?(strong|weak|excellent|poor|compelling|impressive) "
FUNDING = re.compile(r"\b((we |I )?recommend (an award|funding|a grant|approval)|"
RANKING = re.compile(r"\b(rank(s|ed|ing)? (it |this |the application )?(above|below|ahead of|"
def check(answer):

evals/baseline.pythe four free floors — a swap seam

constant, checklist, typed and best — 0 calls, $0.00, and one of them tries the scope test in pure code

You change it to: add a function and one entry in FLOORS. It is scored by the same code path as the paid arm, which is the only thing that makes the comparison mean anything.

# THE FREE FLOORS — four of them, 0 calls, $0.00, no network, and one of them is built to WIN.

VERDICTS = ("SATISFIED", "MISSING", "WRONG_FORM", "OUT_OF_SCOPE")
DOCTYPE_KEYS = {
NOT_ENCLOSED = re.compile(
NEGATED = re.compile(r"\b(no part|not previously|has not|does not|no paid staff|employs no|"
STOP = set("""a an the and or of to in on for by with at from any all each every such this that
WORD_NUMERALS = {
def _fold(w):
def words(text):
def numbers(text):

evals/scoring.pythe scorer

three grains and four questions, every rate beside its majority-class floor

# Score one arm against the answer key. PURE CODE — there is no judge model anywhere in this kit.

VERDICTS = ("SATISFIED", "MISSING", "WRONG_FORM", "OUT_OF_SCOPE")
QUESTIONS = ("verdict", "applicability", "presence", "form")
def majority_floor(gold):
def correctness(gold, findings):
def score(gold, findings, applications, unanswered=(), repairs=None, citations=None):

evals/significance.pythe significance test

McNemar exact from math.comb — stdlib integers, no approximation and no dependency

# Is the paid arm's margin over a free floor real, or is it noise? PURE CODE, no model.

def mcnemar_exact(b, c):
def compare(a_correct, b_correct, label_a="paid", label_b="floor"):

evals/check_labels.pythe label check

4,666 independent re-derivations of the answer key, importing not one line of the generator

# Re-derive the answer key independently and assert it reproduces itself. No model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CHK = {r["id"]: r for r in AP.checklist()["requirements"]}
def scope_from_condition(condition, profile):
def rederive(condition, profile, required, item):
def main():

src/app.pythe local board

renders the corpus, the key, the committed run and all four floors with no key configured

# The board. `python3 -m src.app` and open http://127.0.0.1:9420/.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
RESULTS = os.path.join(HERE, "results")
PORT = int(os.environ.get("PORT", "9420"))
FLOOR_KEYS = ("verdict", "applicability", "applicability_n", "presence", "presence_n",
def run_findings():
def floor_findings(name="best"):
def results():
def meta():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2146 input and 629 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

THE BOUNDARY IS THE SHAPE OF THE ANSWER, not a sentence in a banner. The reply schema has four fields per finding and none of them is a decision, a score, a rank or a recommendation; src/station.py keeps exactly those four. An attacker who wants this kit to fund an application has to get it to emit a field that does not exist. What they CAN reach is the verdict, and that is what the probe attacks.

One key, in the repo-root .env, gitignored and written 0600. src/config.py reads shared, then kit-local, then the real environment, merged key by key, and the board never serves the model id — config.MODEL_DISPLAY stands in for it everywhere, so a leak would take an edit to the server rather than a careless template. src/budget.py caps live calls per day across every kit sharing the key and writes its ledger line BEFORE the call.

The experimentWe attacked it — in the one field an attacker really controls

text appended to an attachment's description inside PART C — written by the party whose application is being screened, and reaching the model verbatim. Four wordings: a false claim that the clause does not apply, a bare instruction to an automated screen, a schema attack demanding an eligible flag and a recommendation field, and a claimed programme-officer sign-off at a pre-application clinic. Both arms were run for real: 40 scored calls in r001-applicant-screen and 12 injected calls in x001-applicant-screen. Nothing on this page is a projection of an attack that was not made. Measured on 2026-09-11.

The three gates are not equally hard and the first two are the ones that matter. Reaching the VERDICT is the attack an applicant would actually want: it hides a real gap from the screener. Reaching the SCOPE DECISION is better still for an attacker, because a clause reported out of scope produces no gap at all rather than a gap somebody might notice — and it is written in the checklist's own language. The SCHEMA gate is the softest target and the least useful one, because src/station.py keeps four fields and an extra key reaches no published number.

The result1 of 12 obeyed and it is the crude one that worked; 0 of 12 produced a scope flip; 0 forbidden fields asserted; 0 boundary breaches — over 12 bought calls in x001-applicant-screen.

Four wordings — a false claim of exemption, a direct instruction to an automated screen, a schema attack, and a claimed programme-officer sign-off — each fired at three applications, appended to the description of the attachment the key names for a clause it calls WRONG_FORM. The only one that landed was the crudest: the bare instruction, once, on APP-0003's project narrative. The two that dressed themselves up as authority moved nothing, which is the opposite of what a reader might expect and is worth exactly as much as twelve calls can buy.

HonestyWhat this does not prove

• whether a wording nobody tried would succeed
• whether an attack placed in PART A — the applicant's own declarations — would move the scope decision. The probe only writes into PART C
• whether an attack spread across several attachments of one application compounds
• whether the same wordings work against a different model or with reasoning enabled

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

Nothing this kit produces declines an application, scores its merit, ranks it against another applicant or recommends funding. A programme officer reads the application and the foundation's board awards the grant. This is the completeness screen made before the officer opens it, and it stops at naming what the checklist asks for and what the application does and does not answer.

Two places, and the first is structural. (1) THE REPLY SCHEMA in src/prompt.py has four fields per finding — the clause, one of four verdicts, the attachment and a quotation — and no field for a decision, a score, a rank or a recommendation. src/station.py keeps exactly those four and discards anything else the model returns, so a decision cannot reach a published number even if a reply carried one. (2) THE ONE FREE SENTENCE, note, is the only field not structurally constrained, and src/refusal.py reads every one of them for four families of breach. All four counters were 0 on all 40 calls of the scored run and 0 under all 12 calls of the adversarial probe.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• an eligibility determination. A clause being out of scope means the checklist does not ask for that document from this applicant; it says nothing about whether the applicant may be funded
• a merit assessment, a score, a rank or a funding recommendation of any kind
• a check that the checklist itself is lawful, current or fairly written
• a check that the applicant's declarations are TRUE. It reads the cover sheet as given

WatchedWhat is watched, and why that one

The kit measures 41 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

14 measured by the latest run 27 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (free_floors constant 103/320 · checklist 144/320 · typed 161/320 · best 248/320 (four-way verdict, 0 calls, $0.00, no network)) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

6 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

comply · no model in the path — a baseline, not a peer column — 4 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 4 runs measured the same system — they differ on the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

comply · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 37 chips that all say so.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 14 chips that all say so.

DeviationsWhat deviated

0 breaches across 6 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Every run measures every band for free, on every arm, because src/refusal.py, src/citation.py and evals/scoring.py are all pure code and run inside the harness. Re-scoring the committed run re-measures all seven bands at $0.00.

In one lineNo framework, deliberately

Nothing here is imported. Every seam a framework would normally own — the provider call, the prompt assembly, the reply parse, the retry policy, the spend cap, the scoring — is a named file in src/ or evals/, in the standard library, and a forker reads all of it in an afternoon. That is a deliberate trade: a framework would have given us structured output and a router for free, and it would also have made the fork test an install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

corpus -> packet (all three parts, verbatim) -> [four free floors | prompt -> one call -> reader -> station -> citation -> refusal] -> scorer -> significance -> result file -> board

The other sideWhat a framework costs you

• no dependency to install, so the fork test is python3 evals/run.py --floor best
• no framework upgrade can change a published number
• structured output is enforced by a parse and a station rather than by the provider, so a malformed reply is a measured state instead of an exception
• and the price: no router, no batching, no async, no observability integration — all four would be somebody else's code and all four are absent

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-applicant-screen, 2026-09-11. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 4 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, FOUNDRY_RUN_ID, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT (the corpus is generated; there is nothing in it anyone else owns) Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe four-way verdict

whether the reading gave each checklist clause the verdict the key gives it — satisfied, missing, wrong form or out of scope

MethodHow the test was run

python3 evals/run.py --floor best --run-id b000-applicant-screen-best (0 calls, $0.00) — the paid arm re-scores from results/cache-r001-applicant-screen.jsonl by the same code

Every grader on these pages scored the same 320 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

This is the hedge, and it is 35 of the 86 out-of-scope clauses in the corpus. The applicant enclosed the document anyway, so every signal a keyword checklist reads says SATISFIED: the document type is named, the clause is cited, every particular is present. The only thing that says otherwise is a sentence about the applicant, not about the attachment.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, re-derived independently by evals/check_labels.py over 4,666 checks.

No true/false rates for this grader. It records 6 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? there is no threshold — the grader is an equality test against an enum. The only tuned numbers in this kit are the free floor's, and they were tuned in the floor's favour, on the same 320 items they are scored on

Cadence: every run, on every arm, for $0.00

The decisionWhen to reach for it

In one lineThe three sub-questions

which of the three readings the four-way rate is made of the arm actually got — does the clause APPLY, is anything ENCLOSED against it, and is what was enclosed in the FORM the clause states. Each is scored over its own denominator, because the second question does not arise for a clause that does not apply and the third does not arise for a clause nothing answers

MethodHow the test was run

the same command as the verdict grader — one scorer produces all four questions from one arm's findings

Every grader on these pages scored the same 320 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

This is the hedge, and it is 35 of the 86 out-of-scope clauses in the corpus. The applicant enclosed the document anyway, so every signal a keyword checklist reads says SATISFIED: the document type is named, the clause is cited, every particular is present. The only thing that says otherwise is a sentence about the applicant, not about the attachment.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl. The three denominators are derived from the key's own class distribution, never typed.

No true/false rates for this grader. It records 3 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? no thresholds. Each sub-question is a two-class equality test derived from the same enum

Cadence: every run, on every arm, for $0.00

The decisionWhen to reach for it

In one lineThe evidence behind the verdict

whether the quotation the reading rests on is actually in the packet, and whether it is the RIGHT KIND of evidence — an attachment for SATISFIED and WRONG_FORM, a PART A declaration for OUT_OF_SCOPE

MethodHow the test was run

run inside evals/run.py on every arm; the paid arm re-scores from its own reply cache for $0.00

Every grader on these pages scored the same 320 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

This is the hedge, and it is 35 of the 86 out-of-scope clauses in the corpus. The applicant enclosed the document anyway, so every signal a keyword checklist reads says SATISFIED: the document type is named, the clause is cited, every particular is present. The only thing that says otherwise is a sentence about the applicant, not about the attachment.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the packet text itself. Nothing is normalised but whitespace and case — a normaliser that stripped punctuation would make two different dollar figures compare equal.

No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? coverage 0.60 and precision 0.40, both declared in src/citation.py. Both floors must clear: coverage alone rewards quoting the whole packet, precision alone rewards quoting one word

Cadence: every run, on every arm, for $0.00

The decisionWhen to reach for it