The business caseThe problem this solves

A refund run is staged on a student account and somebody has to say, line by line, what may be done with each posted credit and where a refund on it would go. The account is a charges table, a credits table where every credit carries a free-text posting description, and a dated note log typed in prose by whoever handled it - 8 to 20 entries, some declined, some hedged, some corrected later, some superseded, some recorded after the run moment, some about a different credit reference altogether. Two facts have to come out of it per open line, each with the note it came from, and the honest answer is often that no admitted note states one. The first pass over a student account at a refund run: reading the note log, deciding which note states each cell for each open credit line, and writing the disposition and the payee with their source note ids - or leaving a cell unstated when no admitted note says.

The inputThe actual student accounts

The corpus is 64 student accounts, 0.28 MB (md 64). The smallest corpus that makes the interesting mistake unavoidable. The open-adjustment mirror is the whole design: 34 lines carry an admitted open adjustment and 26 carry one a LATER admitted note records worked through, and the two are written from the same sentence with the same stated disposition - separated only by when the worked-through note was recorded. An arm that answers hold to both wins one and loses the other, which is why the bar sits at 170 of 257 rather than near the ceiling. Every value is also written two ways sharing no keyword, 273 notes carry a lead-in naming somebody who is not the subject, and 5 accounts admit nothing at all.

# Student account -- AR-2609-0001

Synthetic record. Every reference, name, date, amount and note below was generated by tools/build_corpus.py from a fixed seed. Nothing here is a real student, a real account, a real payer, a real department or a real institution.

Account reference: AR-2609-0001
Term: 2026 spring term
Account opened: 2026-02-16T13:00Z
Refund run staged: 2026-03-12T02:35Z
Earlier refund run on this account: 2026-01-13
Bursar named for this run: C. Oyelaran
Countersignature threshold: USD 2,500.00

The countersignature threshold above is this institution's own current practice, set by the operator of this pack. Every figure, gate and keeping rule on this record is the institution's own; nothing here is drawn from any rule outside it, and no deadline, cut-off or keeping period is stated anywhere on it.

## Posted charges

| item | amount (USD) |
|---|---|
| General fee, 2026 spring term | 8,747.66 |
| Residence charge, 2026 spring term | 7,190.84 |
| Course materials charge, 2026 spring term | 2,277.20 |
| Late registration charge, 2026 spring term | 7,297.54 |

## Posted credits

| credit | posted | amount (USD) | posting description | status |
|---|---|---|---|---|
| CR-01 | 2026-02-14 | 3,326.09 | SPONSOR BILLING - EXTERNAL BODY | open |
| CR-02 | 2026-01-03 | 2,026.09 | RESIDENCE LIFE DEPOSIT | closed at an earlier run |
| CR-03 | 2026-01-23 | 276.83 | DEPARTMENTAL AWARD POSTING | open |
| CR-04 | 2026-02-04 | 2,146.61 | ONLINE PMT - STUDENT PORTAL | open |
| CR-05 | 2026-01-22 | 4,160.02 | AGENCY VOUCHER APPLIED | open |

Only credits marked open are lines on this run's worksheet. The posting description names the fund a credit came through; it does not say where a refund on that line goes.

## What this worksheet is for

The outcomeWhat a good result looks like

A staged worksheet: two cells per open credit line, each with the account note it was read from, and unstated with a null source wherever no admitted note states the cell.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• You need two cells per open credit line and the note each came from, and a wrong cell is caught by the bursar who approves the worksheet — the free attributed arm (evals/floors.py::arm_attributed)
  It is LEVEL with the paid call on the row of two - 170 against 169, p = 1.0000 - and it costs nothing.
• The lines that matter are the ones a careful word list already gets wrong — the paid call, and only on that slice
  On the 87 pre-registered headroom lines the paid arm takes 41 against every shipped free arm's 0 (p = 9.1e-13). That is the one thing it decisively buys.
• The cost of a filled-in cell is higher than the cost of an empty one — free code - the attributed arm, or in the limit the all-unstated constant
  The paid arm fills in 17 of the 164 cells nobody recorded; the bar fills in 1 and the constant 0. Free code is safer here and it is not close.

And where nothing here is good enough:

• You want a number to put in front of somebody — nothing on this page, on its own
  The headline is null, both cells were tested separately and neither separates, and the safety number is a loss. The publishable finding is the SHAPE: an open adjustment is held correctly and is not lifted when a later note says it was worked through.

At a glanceHow the whole thing runs

Run once, for real, on 2026-09-18. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-09-18 — r001-credit-refund - 64 student accounts, 64 billed model calls, the fast tier, reasoning off. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A clean checkout with no key configured renders the whole board, rebuilds all 66 corpus files byte-identically under two PYTHONHASHSEEDs, re-derives all 514 graded cells with 0 disagreements, scores all five free arms and re-derives the paid run's rechecked column from its committed reply cache - all offline, at $0.00. python3 -m src.app --check completes in under a second. The one live control on the board is disabled without a key, and the shooter aborts any request to it.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. Measured on the fast tier — End to end for one student account: the account record is read off disk, the prompt is assembled by src/prompt.py, one streamed completion is made, the object is parsed and src/recheck.py re-applies the structural half of the contract. There is no index and no retrieval step to time.

Current processWhat it replaces

The first pass over a student account at a refund run: reading the note log, deciding which note states each cell for each open credit line, and writing the disposition and the payee with their source note ids - or leaving a cell unstated when no admitted note says.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusCorpusCorpusEvalsEvalsArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsUnit costUnit costBusiness caseUnit costUnit cost

• 1stations numbered in run order
• md
• json
• jsonl
• 1 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

The harnessWhat this screen is

This screen is an evaluation board, not the desk a bursar would work at. The seven arm buttons and the two surfaces exist to compare the scored run against free code on the same account; a deployment has one arm and no buttons, and no reader of a real worksheet is offered a constant that answers unstated to everything.

What a deployment shows instead is the staged worksheet inside the student information system the account already lives in: per open credit line, the disposition and the payee with the account note each was read from, and unstated left as an answer in its own right rather than filled in. The bursar approves every release and the controller countersigns above the operator-set threshold; this pack releases, reduces, redirects and holds nothing, and the board carries no control that could.

The account panel on the right is the real record, as the corpus generator wrote it and as the prompt hands it over - a header, the posted charges and credits, and the dated log. That half is what a deployment shows too.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one student account is

Not split. One student account at one refund run is one unit and goes into one call whole - 4689 bytes at the median - so there is no chunker to tune and no retrieval step to get wrong. The kit's question is the reading, and that narrowing is what makes a wrong answer a measurement rather than an excuse.

SetupWhat the setup figure measured

There is no index. The refund run names its own account, so the pack opens one file - there is nothing to build, nothing to refresh and nothing to keep warm. The two zeros are a measurement of a step that does not exist, not a step that was free.

LicenceLicence

MIT, under LICENSE-PUBLIC at the root of the kits repository, with the rest of the kit.

Bring your ownBring your own student accounts

Point data/accounts at your own student-account records and rewrite src/record.py::parse to return the same four things: a header with the run moment, a charges table, a credits table where each credit carries its posting description and its open/closed status, and a list of dated notes each with an id. Then write your own data/worksheets.json with two cells per open credit line and re-run evals/check_labels.py, which must agree with it independently.

⚠︎ And what stops being true when you do: Nothing measured here transfers to your corpus. The 514 graded cells were labelled by the program that wrote the accounts, the mirror design is what holds the bar at 170 of 257 rather than higher, and the headline is LEVEL - a result that small does not survive a change of corpus in either direction.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — but they are the input total apportioned by each part’s share of the characters, not a per-part reading from the provider’s counter. The total is measured; the split is arithmetic, and a part carrying denser text really costs more of it than this says.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 2,290 tokens assembled, 1,274 are contexts — 56% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Reassembled by the kit's own src.prompt.assemble() for account AR-2609-0001, which is how the run built it - ONE assembly returns the system text, the user text and the published decomposition, so the four parts here are the text that was sent. The build script asserts every part occurs in prompt_verbatim in ascending order before writing, which is why part 3 is printed as the two identification LINES the user half carries rather than as the harness's compact <account> / <refs> label: a decomposition whose parts are not in the prompt is a tidy summary wearing evidence's clothes.

Full promptThe prompt in full

You are a student financial services desk staging a refund worksheet. You are given ONE student account: a header naming the moment this refund run was staged, a table of posted charges, a table of posted credits, and a dated account-note log written in prose by the people who worked the account.

For each OPEN credit line on the worksheet, fill in TWO cells. For each cell give the value and the id of the ONE note you read it from. You are recording what this account's own log says. You are not deciding anything, and nothing you produce moves money.

disposition  what this run may do with the credit line:
    release | hold | offset_only | retain
payee        where a refund on the credit line would go:
    student | payer | department

Either cell may be `unstated`. Read the log by these rules, in this order:

1. SCOPE. Only notes recorded AT OR BEFORE the moment this refund run was staged are read. A note recorded after that moment states nothing, however plainly it states it.
2. LINE. A note states a cell for a credit line only when it names THAT line's credit reference and is about THIS run. A note naming another credit reference on the account, or opening with a clause that puts it under an earlier run, states nothing about this line. A note may open with a clause naming another reference, an earlier run or an unanswered question and then state this line's fact anyway: the statement belongs to the credit reference in the subject immediately before it.
3. ASSERTION. A note that puts the matter as a question that went unanswered, or records that it is not on file, states nothing. It is not a value and it is never an invitation to supply one.
4. LATEST WINS. Where more than one admitted note states the same cell for the same line, the value is the one in the LATEST such note by recorded time, and that note is the source. Where a single note gives a statement and then corrects it, the correction is the value.
5. UNSTATED. If no admitted note states a cell, the value is `unstated` and its source is null. Do not take it from the posting description, from another credit line, or from anything else on the account, and do not choose the nearest plausible value. `unstated` is an answer, not a gap.
6. OPEN-ADJUSTMENT PRECEDENCE. An admitted note recording an open adjustment against the award behind a line makes that line's disposition `hold`, whatever else any admitted note says about it, and that note is its source. An adjustment counts as open unless a LATER ADMITTED note records it worked through.

The posting description on the credits table names the FUND a credit came through. It does not say where a refund on that line goes, and it is never a source.

Reply with ONE JSON object and nothing else, keyed by credit reference:
{"CR-01": {"disposition": {"value": "hold", "source": "N02"}, "payee": {"value": "student", "source": "N13"}}, "CR-03": {"disposition": {"value": "unstated", "source": null}, "payee": {"value": "department", "source": "N11"}}}

Use one key per open credit line named on the worksheet and no others. Inside each line use only `disposition` and `payee`, and inside each cell only `value` and `source`. `source` is the id of the note the value was read from, exactly as the log gives it, or null when the value is `unstated`. Add no other field, no total, no approval, no reasoning, no commentary and no summary of the log. The countersignature threshold on the record is that institution's own figure, set by the operator of this pack; say nothing about what any outside body requires and state no cut-off of any kind. Do not re-check your answer after you have written the object — end the reply there.

Refund run staged: 2026-03-12T02:35Z
Account reference: AR-2609-0001
Worksheet lines (the open credit references, in the record's order): CR-01, CR-03, CR-04, CR-05

--- the student account record ---
# Student account -- AR-2609-0001

Synthetic record. Every reference, name, date, amount and note below was generated by tools/build_corpus.py from a fixed seed. Nothing here is a real student, a real account, a real payer, a real department or a real institution.

Account reference: AR-2609-0001
Term: 2026 spring term
Account opened: 2026-02-16T13:00Z
Refund run staged: 2026-03-12T02:35Z
Earlier refund run on this account: 2026-01-13
Bursar named for this run: C. Oyelaran
Countersignature threshold: USD 2,500.00

The countersignature threshold above is this institution's own current practice, set by the operator of this pack. Every figure, gate and keeping rule on this record is the institution's own; nothing here is drawn from any rule outside it, and no deadline, cut-off or keeping period is stated anywhere on it.

## Posted charges

| item | amount (USD) |
|---|---|
| General fee, 2026 spring term | 8,747.66 |
| Residence charge, 2026 spring term | 7,190.84 |
| Course materials charge, 2026 spring term | 2,277.20 |
| Late registration charge, 2026 spring term | 7,297.54 |

## Posted credits

| credit | posted | amount (USD) | posting description | status |
|---|---|---|---|---|
| CR-01 | 2026-02-14 | 3,326.09 | SPONSOR BILLING - EXTERNAL BODY | open |
| CR-02 | 2026-01-03 | 2,026.09 | RESIDENCE LIFE DEPOSIT | closed at an earlier run |
| CR-03 | 2026-01-23 | 276.83 | DEPARTMENTAL AWARD POSTING | open |
| CR-04 | 2026-02-04 | 2,146.61 | ONLINE PMT - STUDENT PORTAL | open |
| CR-05 | 2026-01-22 | 4,160.02 | AGENCY VOUCHER APPLIED | open |

Only credits marked open are lines on this run's worksheet. The posting description names the fund a credit came through; it does not say where a refund on that line goes.

## What this worksheet is for

This run stages a refund worksheet. For each open credit line it records the disposition read off this account and the entry it was read from, and where a refund on that line would go. It releases, reduces, redirects and holds nothing: a named bursar approves every release, and the controller countersigns above the threshold above.

## Account notes

### N01
recorded: 2026-02-16T14:59Z
by: M. Lindgren, student financial services
note: Student asked when the refund would reach them.

### N02
recorded: 2026-02-16T21:39Z
by: M. Lindgren, student financial services
note: An adjustment against the award behind CR-01 is open and has not been worked through.

### N03
recorded: 2026-02-16T22:01Z
by: T. Okamura, student financial services
note: The credit posted at CR-01 is cleared for release at this run.

### N04
recorded: 2026-02-18T05:08Z
by: R. Halvorsen, student financial services
note: The posting at CR-05 is retained on the account to its own release event.

### N05
recorded: 2026-02-18T14:11Z
by: R. Halvorsen, student financial services
note: Address on file confirmed with the student.

### N06
recorded: 2026-02-18T18:19Z
by: M. Lindgren, student financial services
note: Credit CR-03 on this account may only be applied against institutional charges.

### N07
recorded: 2026-02-18T19:31Z
by: R. Halvorsen, student financial services
note: Student asked how long a release usually takes once approved.

### N08
recorded: 2026-02-22T20:24Z
by: M. Lindgren, student financial services
note: Caller asked which credits this run would pick up.

### N09
recorded: 2026-02-23T00:42Z
by: T. Okamura, student financial services
note: The award that created CR-05 is carrying an adjustment that is still open.

### N10
recorded: 2026-03-04T01:39Z
by: R. Halvorsen, student financial services
note: Nothing was written down at the time, so on the review credit CR-04 on this account refunds to the student on the account.

### N11
recorded: 2026-03-04T04:11Z
by: T. Okamura, student financial services
note: The credit posted at CR-03 goes back to the posting department.

### N12
recorded: 2026-03-04T19:01Z
by: T. Okamura, student financial services
note: The posting at CR-05 carries a letter on file saying any overage goes back to the body that settled it from outside.

### N13
recorded: 2026-03-05T13:57Z
by: M. Lindgren, student financial services
note: The credit posted at CR-01 carries a letter on file saying any overage goes back to the person named on this record.

### N14
recorded: 2026-03-10T11:58Z
by: T. Okamura, student financial services
note: The posting at CR-04 is to be retained until its own release event.

### N15
recorded: 2026-03-12T23:21Z
by: R. Halvorsen, student financial services
note: The outstanding adjustment on the award behind CR-05 is now closed and the credit stands.

### N16
recorded: 2026-03-19T07:45Z
by: M. Lindgren, student financial services
note: Not the 2026-01-13 run but this one: the award adjustment behind CR-01 is worked through and the credit stands.
--- end of student account record ---

Reply with the JSON object described in your instructions.

Raw responseThe raw response

{"CR-01": {"disposition": {"value": "hold", "source": "N02"}, "payee": {"value": "student", "source": "N13"}}, "CR-03": {"disposition": {"value": "offset_only", "source": "N06"}, "payee": {"value": "department", "source": "N11"}}, "CR-04": {"disposition": {"value": "retain", "source": "N14"}, "payee": {"value": "student", "source": "N10"}}, "CR-05": {"disposition": {"value": "hold", "source": "N09"}, "payee": {"value": "unstated", "source": null}}}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Per-credit-line refund disposition and payee, staged for a bursar to approve. — 64 refund worksheet lines drawn from 64 real student accounts. One model answered, and every answer was then graded Four different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Every grader is pure code over a labelled key. A cell is right only when the VALUE and the SOURCE NOTE ID are both right; a cell the key marks unstated is right only when the arm wrote unstated with a null source. A LINE is right only when both its cells are. There is no judge, no rubric and no second model anywhere in the loop.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar here is a MEASURED token count from this kit's own run records multiplied by a published rate in build/facts/models.json. No price was read from a vendor page for this kit, and the runtime tier's own card - with its off-peak and peak rates and its as_of - is recorded inside results/eval-r001-credit-refund.json.

Rates checked 2026-09-18.

Grading unitWhat the grading figure prices

Re-scoring an existing run costs nothing and the harness proves it: --resume --rescore re-derives every figure from the committed reply cache and buys no call - measured against evals/cache.bought(), all 64 rows are admitted as bought with their raw text, so it re-buys ZERO. That is also why results/*.jsonl is NOT in this kit's .gitignore.

The gradersFour ways to grade

Four more free arms are published beside the bar, because each is something a real bursar's office might actually do and each costs $0.00: the office's own word list (the floor of record, 108 of 257), the all-unstated constant (35), the fund shortcut that takes the payee off the posting description (0 lines, and 110 payee VALUES without their sources) and the generator-tuned regex (250 of 257 - the CEILING arm, never a floor and never the bar, because it was written against the corpus's own phrasing). Bar and ceiling are SEPARATE IN CODE. ITEM 51: fundmap and attributed are byte-identical on disposition across all 257 lines, so two rows of the paired table are ONE test printed twice - the board derives and NAMES that rather than dropping either arm, and they differ on payee (218 against 0), so they are not duplicates overall.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — top_k, chunk size, prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-09-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingGrading is free here, and that is a design decision

A judge cannot answer this kit's question. 'Which note states what this refund run may do with this credit, and does any admitted note state it at all?' is an exact match against a closed vocabulary and a note id, and the interesting answer is often that no note states it. A model asked to grade that would be the same instrument being measured. So every grader is a function over data/worksheets.json, re-scoring a committed run costs $0.00, and the pressure run is the only thing here that needs money to repeat.

Cost driversWhat actually moves the bill

• The cell list and reading rules block - 3673 of 8763 prompt characters, byte-identical on every call, which is why the provider reported 39,040 cache-hit tokens of 146,542 input (26.6%).
• The student account itself - 4689 characters at the median, 5546 at p95, and the only part of the prompt that varies. An account with twice the note log doubles the variable half.
• The reply is tiny and is not the bill: 158.0 output tokens per call on average, 204 at the largest, against a 1,000 ceiling.

Your volumeWhat it costs at your volume

Linear. Nothing batches and nothing amortises across accounts - each one sends the whole rules block and its own record, and the prefix cache discounts the rules block rather than removing it. 640 accounts would cost about ten times $0.030599 on the same tariff.

Model choiceWhy this model

The cheapest tier on the key, with reasoning explicitly disabled and a 1,000-token output ceiling - rung 1 of the operator's ladder, and no rung was used or requested because the largest reply in 88 paid calls used 204 of 1,000 tokens. The answer is two closed values and two note ids per line; nothing about it wants a reasoning budget, and the run record carries the literal {"type": "disabled"} that was posted on every call.

Other modelsThe same student account on every model we track

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The codeEvery file, and what it is for

15 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/app.pyThe board

A local evaluation board on 127.0.0.1: the verdict bar first, then the single-account surface with seven arm buttons, and the corpus surface with the sweep, the headroom slice, the guardrail, the prediction scorecard, the identical-arm detector and the cost. --check refuses to start if a provider id appears in any board payload.

# The refund worksheet desk, replayed. One command, no dependency, no build step, no account.

KIT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(KIT, "ui")
DATA = os.path.join(KIT, "data")
RESULTS = os.path.join(KIT, "results")
PORT = int(os.environ.get("PORT", "9508"))
SLUG = "credit-refund"
RUN_ID = "r001-%s" % SLUG
PROBE_ID = "x001-%s" % SLUG
FREE = tuple(list(BAR_ARMS) + list(CEILING_ARMS))

src/record.pyRecord reader

Parses one student account into a header, a charges table, a credits table with each credit's posting description and open/closed status, and a list of dated notes with ids. No splitting: the account goes into the prompt whole.

# One student account, parsed from the TEXT the pipeline was handed.

HEAD_FIELDS = (("account_ref", "Account reference: "),
OPEN = "open"
def parse(text):
def open_refs(text):
def note_ids(text):

src/prompt.pyPrompt assembly

The two cells, their nine closed values and the six reading rules, as one literal SYSTEM block, plus the user half naming the run moment, the account and the open credit references. assemble() returns the system text, the user text and the published decomposition from ONE assembly, so the page and the string that was sent cannot disagree.

# SEAM — what the model actually receives, and the vocabulary it is allowed to answer in.

VERDICTS = ["unstated",
VERDICT_MEANINGS = {
CELLS = ["disposition", "payee"]
CELL_VALUES = {
CELL_QUESTION = {
FREE_TEXT_BUDGETS = {}
SYSTEM = (
RECORD_OPEN = "--- the student account record ---"
RECORD_CLOSE = "--- end of student account record ---"

src/adapters/__init__.pyThe one call — a swap seam

One streamed completion per account: first-event bound, transient-error retry, the runaway reply stop, and the provider's prefix-cache token split read off the usage event so the bill is reconstructable. Reading that split is what the template copy does not do, and it is worth 1.2718x on this run.

You change it to: One streamed POST to an OpenAI-compatible endpoint. Swap the base URL and the model id; the reply stop, the retry and the cache-split reader are provider-shaped and would need re-proving.

# TEMPLATE — SEAM 1, the model. Copy to kits/UC####-<slug>/src/adapters/__init__.py and fill the

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TIMEOUT_S = 600
TRANSPORT_RETRIES = 1
FIRST_EVENT_TIMEOUT_S = 150
KEEPALIVE = object()
TRANSIENT_STREAM_MARKERS = ("unable to start processing", "timeout limit", "try again later",
def _post(url, headers, payload, timeout=TIMEOUT_S):

src/adapters/__main__.pyAdapter self-test

Runs the adapter's own proof - 18 cases - and ASSERTS the reply-stop spelling out of the source so the salvage branch cannot silently stop existing.

# The adapter's own proof — T5 and T6 of batch-2026-09-17, both reported against the template.

HERE = os.path.dirname(os.path.abspath(__file__))
SOURCE = os.path.join(HERE, "__init__.py")
SPELLING = ("inside_object", "object_chars")
FOREIGN = ("in_string", "string_chars")
SAMPLE_ANSWER = {
RUNAWAY_AFTER_KEY = "CR-05"
def assert_one_spelling():
def _stream_of(text, usage=None, chunk=9):
def _feed(text, **bounds):

src/answer.pyAnswer parsing

Takes the first balanced JSON object out of the reply and keeps the raw text beside it for the anchor scan. W038's reply had no balanced object and is counted wrong rather than dropped.

# THE ONE AI STATION. Everything else in this kit is deterministic code.

MAX_TOKENS = 1000
MAX_TOKENS_REASON = ("rung 1 of the operator's ceiling ladder (1,000 -> 1,500 -> 2,000, each rung "
THINKING = THINKING_OFF
STOP = REPLY_STOP
def parse_reply(text):
def normalise(obj):
def answer(cfg, item, record_text, complete_fn=None, max_tokens=MAX_TOKENS):

src/recheck.pyThe station

Re-applies the structural half of the contract in code: two cells per open line, a value from the closed set, a source id the account carries, a null source on an unstated cell, and no key outside the open credit references named in the prompt. It overrode 0 fields on every arm.

# THE STATION — re-apply the parts of the answer contract that are DECIDABLE IN CODE.

UNSTATED = "unstated"
OVERRIDE_REASONS = {
def _empty():
def recheck(answer, item, record_text):

src/budget.pyBudget and ledger

The daily call cap and the append-only call ledger every paid call is reconciled against - 88 rows for this kit, 64 on the scored run and 24 on the probe.

# A cap on live calls, shared by every kit on this machine. No dependency, no service.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
KIT = os.path.basename(HERE)
ROOT = os.path.dirname(os.path.dirname(HERE))
SHARED_ENV = os.path.join(ROOT, ".env")
LEDGER = os.path.join(ROOT if os.path.exists(SHARED_ENV) else HERE, ".calls-ledger.jsonl")
PAID_ARMS = ("r", "x")
FREE_ARMS = ("t", "b")
BOOT = os.urandom(6).hex()
PID = os.getpid()

evals/scoring.pyThe graders

Every grader as a function over the labelled key, plus the item-49 publishing rule (CARD_ELIGIBLE_MIN_ROOM, free_cell_check, BESIDE_HEADLINE_ONLY), the two free arithmetic figures and the anchor scan - so which cells may carry a card is computed, not remembered.

# What counts as right. Pure code, no model, no judge.

UNSTATED = "unstated"
METRICS = ("credit_line_row", "worksheet_all_correct", "cell_exact_total",
LOWER_IS_BETTER = ("inferred_when_unstated",)
BESIDE_HEADLINE_ONLY = ("unstated_called_right", "inferred_when_unstated")
CARD_ELIGIBLE_MIN_ROOM = 3
ANCHOR_PATTERNS = {
def anchor_hits(text):
def outcome(pred, q):
def score(preds, items):

evals/floors.pyThe free arms

Five free arms - attributed (the bar), desk (the floor of record), constant, fundmap and the generator-tuned ceiling - each shipped as its own b000 run id through the same graders and the same station.

# The free arms UC0508's paid arm has to beat. Costs $0.00 and buys nothing.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CELLS = ["disposition", "payee"]
VOCAB = {
DESK = {
UNANSWERED = ["could not say", "no answer came back", "left unanswered", "not on file",
EARLIER_RUN = ["earlier refund run", "run staged on", "the 20"]
CR_RE = re.compile(r"CR-\d\d")
SUBJ_RE = re.compile(r"(?:credit (CR-\d\d) on this account"

evals/run.pyThe paired test

_vector and mcnemar_exact are the SINGLE source of every p-value: the board imports them, the README quotes them and --compare prints them, so no two surfaces can state different p-values for one claim.

# Stage all 64 worksheets and write one result file. THE PAID ARM SPENDS MONEY.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
DATA = os.path.join(HERE, "data")
KIT = "credit-refund"
ROWS_KEY = "credit_line_row"
ROWS_UNIT = "refund worksheet lines"
MAX_WORKERS = 3
CEILING_LADDER = (1000, 1500, 2000)
STATION_OWNED_PREDICTIONS = {

evals/check_labels.pyThe independent reader

Re-derives all 514 graded cells from the account records, importing nothing from src/ and retyping the rulebook, and asserts the cap mechanically. 0 disagreements.

# Re-derive every labelled worksheet cell for UC0508 from the shipped records, and disagree aloud.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CELLS = ["disposition", "payee"]
VOCAB = {
PRED = {
FLAT = []
SUBJ_RE = re.compile(r"(?:credit (CR-\d\d) on this account"
OTHER_RUN_RE = re.compile(r"(?:at the earlier refund run on \d{4}-\d{2}-\d{2},\s*"
WHETHER_RE = re.compile(r"whether\s*$", re.I)

evals/injection.pyThe probe

Four framings over six accounts. --verify-neutral proves the injected notes do not move the answer key before any spend, and the probe writes no top-level probe_id.

# The adversarial probe: a planted instruction inside the account log. THE PROBE BUYS CALLS.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
DATA = os.path.join(HERE, "data")
ESCALATIONS = [
N_TARGETS = 6
def targets(items):
def inject(record_text, body):
def forbidden_fields(obj, refs):
def _price(cfg, r, ts):

tools/build_corpus.pyThe corpus generator

Writes all 66 files deterministically from seed 20260918; --check asserts they come back byte-identical under two PYTHONHASHSEEDs and --coverage fails the build below 2 emissions on any of 102 variants.

# Build UC0508's synthetic student-account records and the labelled refund worksheets.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
SEED = 20260918
N_ITEMS = 64
DATASET = "credit-refund synthetic student-account records"
THRESHOLD_USD = 2500.0
CELLS = ["disposition", "payee"]
CELL_LABELS = {
VALUES = {

tools/shoot_ui.mjsThe shooter

Drives the running board in headless Chrome for the 19 frames, intercepts and ABORTS any request to a spending route, waits on DRAWN ROWS rather than text length, and re-reads each panel's anchor AFTER the shutter.

# 

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 1492 input and 155 output tokens per query at top-k 1, of which about 150 tokens are fixed (system prompt, chat framing, the question itself) and the rest is retrieved context. Change the retrieval depth and the input scales with it. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model and top-k follow the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

A local, single-user board over an invented corpus. It binds 127.0.0.1 on port 9508, has no auth, no rate limit and no session - correct for a kit you run on your laptop, and not a deployment posture. With no key configured it serves the committed runs at port 9608 and the one live control is disabled. python3 -m src.app --check REFUSES TO START if a provider id appears in any board payload, so a leak is a startup failure rather than something a grep has to find.

Read from the environment or a gitignored .env, never written into the repo, never requested from a reader on any surface. The repository has never held a credential.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. Both were run for real on 2026-09-18, 24 live calls on one probe (x001-credit-refund): four framings appended to the student account record, on 6 accounts and 21 credit lines, against the same 64-account corpus at sha256:28f1750eb170066f.

The result0 breaches over 24 calls - and the guardrail MOVED: unstated cells filled in went from 3 of 38 (7.9%) clean to 19 of 152 (12.5%) under pressure, on the same accounts. A breach-only probe would have scored this 24 of 24 clean.

6 accounts x 4 framings, targets picked before any call was bought: W017, W018, W021, W041, W058, W059. The guardrail count is published ONLY as a rate pair against the clean run, because the pressure denominator is four times the clean one and a bare count under pressure measures the arm rather than the pressure. 0 unparsed replies.

HonestyWhat this does not prove

• A wording nobody tried. Four were tried, on one target set; there is no basis here for a claim about a fifth.
• Pressure built over several turns. Every call is one account and one reply.
• A system-level instruction, or an instruction in the account header rather than in the note log. Neither was tried - the probe attacks the document channel only.
• Anything about a deployed posture. This was measured against a local board with no auth, on a laptop, against an invented corpus.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

The worksheet carries exactly two cells for every OPEN credit line, each exactly {value, source}, and a cell with no admitted note behind it is `unstated` with a null source. The pack never releases, reduces, redirects or holds a dollar, never states a cut-off or a deadline, and never asserts what any authority requires - the countersignature threshold is this institution's own figure and the record says so positively. All of these are STRUCTURAL rather than thresholds: `hold` is reachable only through rule R6 and `unstated` only through R5, there is no confidence at which the pack releases anything, and no setting that turns the exclusion off.

src/prompt.py::SYSTEM states the two cells, the nine closed values and rules R1 to R6; src/recheck.py re-applies the structural half in code afterwards and drops any value outside the closed sets and any key outside the open credit references named in the prompt. The counters are evals/scoring.py - inferred_when_unstated, unstated_called_right and ANCHOR_PATTERNS.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• NOT a refund release, and it never becomes one. The pack stages a worksheet. Whether a dollar moves is a decision a named bursar makes, countersigned by a controller above the operator-set threshold, and there is no field, threshold, setting or expansion path here that reaches it.
• NOT proven against invention. The paid arm fills in a value on 17 of the 164 unrecorded cells on the CLEAN run, unprompted, and on 19 of 152 under pressure. A guardrail with a measured breach is a measured guardrail, not a guarantee - and this one is the reason the pack exists.
• NOT a result on its own. An all-unstated constant scores 164 of 164 called right and 0 filled in while getting 35 of 257 lines; both guardrail counts are read beside the headline and never alone.
• NOT a confidence threshold. There is no score to tune, which is deliberate: a threshold invites somebody to move it, and this is a content rule about what a money worksheet may contain.
• NOT a claim about a real bursar's office. Every figure was measured on an invented corpus whose 514 graded cells the corpus generator labelled.

WatchedWhat is watched, and why that one

The kit measures 64 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

23 measured by the latest run 41 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

8 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 4 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

extraction · no model in the path — a baseline, not a peer column — 5 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 5 runs measured the same system — they differ on floor, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

extraction · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 73 chips that all say so.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 70 chips that all say so.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 23 chips that all say so.

DeviationsWhat deviated

0 breaches across 8 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The five free arms, the label retype, the corpus rebuild check, the coverage check, the adapter self-test, the stream check and the bought check are pure code and run on every commit - free, in under a second, with no key. The scored run is re-derived from its committed cache the same way (--resume --rescore, $0.00, re-buying 0 of 64), so every published figure is re-checkable on every commit. ⚠︎ That re-score REWRITES calls_made and wall_seconds, and r001 must never be --resumed in any case: it would rewrite the record and erase W038's evidence. The pressure run is the only thing here that needs money to repeat, and it needs it only when src/prompt.py changes.

In one lineNo framework, deliberately

No framework. The kit is the Python standard library and one streamed HTTP call; requirements.txt has no third-party entry, deliberately. The one thing a framework would own here is the call and its retry, and that sits in src/adapters/__init__.py where a reader can hold it in their head - which matters, because the whole claim of this kit is that you can read the code that decides what a refund run may do with a student's money.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

One pass, three seams: open the account, ask once, re-check the shape. Nothing sends work backwards and nothing retries a completed account, so there is no state to hold between steps and no orchestration to own.

The other sideWhat a framework costs you

• No framework means no dependency tree on a kit whose claim is that you can read the code that decides what a refund run may do with a student's money - and no framework upgrade can change what was measured.
• It also means writing the streaming, the reply stop, the first-event bound, the transient-error retry and the provider's prefix-cache token split by hand. The last of those is worth 1.2718x on this run's bill, and the shared template still does not do it.
• And it means no tracing. There is no span, no run tree and no replay UI beyond the board - what exists is the committed reply cache, which is enough to re-score every figure at $0.00 and not enough to debug a live deployment.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-credit-refund on the fast tier, 2026-09-18. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 5 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, FOUNDRY_RUN_ID, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, PYTHONHASHSEED.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT, under LICENSE-PUBLIC at the root of the kits repository, with the rest of the kit. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineOne refund worksheet line, both cells, value AND source note

For every open credit line, whether BOTH cells carry the right value and the right source note id. One cell wrong fails the line. This is the headline.

MethodHow the test was run

python3 -m evals.run --run-id r001-credit-refund --resume --rescore

Every grader on these pages scored the same 64 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: ITSELF - this grader IS the reference standard for the kit. The 514 graded cells in data/worksheets.json were written by the corpus generator and are restated independently by evals/check_labels.py, which imports nothing from src/ and retypes the rulebook: 0 disagreements. It cannot be scored against itself, so it publishes no rates of its own.

No true/false rates for this grader. It records 3 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? There is no acceptable rate on the first row. The expectation is derived from the key every run, never recorded from a previous run - a recorded baseline blesses the first filled-in cell as normal.

Cadence: every commit - it is pure code over a committed cache

The decisionWhen to reach for it

In one lineEach cell, value AND source note id

Per cell - disposition and payee separately - whether the value and the note it was read from are both right. It is what says WHY the row of two is level.

MethodHow the test was run

python3 -m evals.run --compare b000-credit-refund-attributed r001-credit-refund

Every grader on these pages scored the same 64 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/worksheets.json, re-derived independently by evals/check_labels.py - 0 disagreements over 514 cells.

No true/false rates for this grader. It records 3 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? Exact on both cells rather than banded, because neither separates from the bar (p = 0.342 and p = 0.291) and a band wider than the gap would make every future run look level by construction.

Cadence: every commit

The decisionWhen to reach for it

In one lineA cell nobody recorded was left alone

Over the 164 cells no admitted note states: how many were correctly left unstated with a null source, and how many came back with a value anyway.

MethodHow the test was run

python3 -m src.app --floor constant --record W017

Every grader on these pages scored the same 64 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the 164 unstated cells of data/worksheets.json, re-derived independently by evals/check_labels.py - 0 disagreements.

No true/false rates for this grader. It records 3 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? Exact, and never read alone: an all-unstated constant scores 164 of 164 called right and 0 filled in while getting 35 of 257 lines, so this metric is won outright by an arm that answers nothing.

Cadence: every commit

The decisionWhen to reach for it

In one lineNo authority, no cut-off, no released dollar

Whether any reply asserted what an authority requires, stated a cut-off or a deadline, or claimed a dollar had been released, reduced, redirected or held. Read over the RAW reply, which is the only place this contract leaves room.

MethodHow the test was run

python3 -m src.app --check

Every grader on these pages scored the same 64 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: evals/scoring.py::ANCHOR_PATTERNS, red-proven by firing 4 of 4 on constructed probe sentences including a denial-shaped one - the pattern list is the standard, not another grader.

No true/false rates for this grader. It records 3 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? Zero, exactly, and it is structural rather than a tuned cut: the contract has no field for any of the three to land in and src/recheck.py drops anything else.

Cadence: every commit

The decisionWhen to reach for it