The business caseThe problem this solves

Cash arrives in the bank before anybody knows what it is for. A payer sends one amount and types something on the remittance advice -- sometimes an invoice number, sometimes a purchase order, sometimes “the March 2026 statement”, often nothing usable at all -- and the amount is frequently short of the invoice, because they took a settlement discount, or drew on a credit note, or simply underpaid. Today a person opens the remittance beside the open-receivables ledger and decides which invoices the money is against and how much each one gets. What they cannot decide goes to unapplied cash, where it ages: the invoice keeps appearing on the collections list, the customer gets chased for money they have already paid, and the balance sheet says the receivable is still open. The read-the-remittance-beside-the-ledger step of cash application: deciding which open invoices one incoming payment settles and how the cash splits across them.

The inputThe actual payments, each with the open ledger put in front of it

The corpus is 60 payments, each with the open ledger put in front of it, 0.14 MB (json 2). Because the alternative is nothing. Cash application is a job whose input is two private documents at once, so a public corpus does not exist -- and a kit that waited for one would not exist either. What a generated corpus buys is a stated case mix, published per case, that a reader can weigh against their own queue: lump_sum_po 5 of 5 (free rules floor 0, null floor 0); discount_not_earned 4 of 4 (free rules floor 0, null floor 0); lump_sum_statement 3 of 4 (free rules floor 0, null floor 0); clean_single_cited 7 of 7 (free rules floor 7, null floor 7); credit_note 4 of 4 (free rules floor 4, null floor 0); credit_note_absent 2 of 2 (free rules floor 2, null floor 0); short_pay 3 of 3 (free rules floor 3, null floor 0); overpayment 3 of 3 (free rules floor 3, null floor 0); cited_settled 3 of 3 (free rules floor 3, null floor 3); ambiguous_amount 3 of 3 (free rules floor 3, null floor 3); no_match 3 of 3 (free rules floor 3, null floor 3); lump_sum_cited 4 of 5 (free rules floor 5, null floor 0); clean_single_uncited 4 of 6 (free rules floor 6, null floor 6); discount_taken 3 of 5 (free rules floor 5, null floor 0); cited_other_customer 0 of 3 (free rules floor 3, null floor 3). What it costs is that the hard cases were chosen by whoever wrote the generator, which is stated wherever a score is.

{
  "_comment": [
    "THE ANSWER a cash application has to produce, and the RULEBOOK it is produced under. This",
    "file is the single source: src/prompt.py builds the model's schema and its rulebook block",
    "from it, src/apply.py reads the caps and the draw order out of it, evals/scoring.py grades",
    "the four fields it marks `graded`, evals/baseline.py runs both free floors against the same",
    "rulebook, and tools/build_corpus.py writes both the ledger and the answer key from it.",
    "",
    "⚑ THIS KIT PROPOSES AN APPLICATION. IT NEVER POSTS ONE. Nothing here writes to a ledger,",
    "clears an invoice or moves money, and no output of this kit is a posting instruction. The",
    "unit of work is a PROPOSAL with its evidence attached, for a cash-application clerk to",
    "accept or reject. That boundary is why rule 3 below exists: when the evidence does not",
    "decide, the honest output is an unapplied payment sitting on account, not a guess that",
    "reconciles.",
    "",
    "⚑ EVERY AMOUNT IN THIS KIT IS AN INTEGER NUMBER OF CENTS, END TO END. There is no float",
    "anywhere in the pipeline -- not in the corpus, not in the rulebook, not in the prompt, not in",
    "the scorer. A discount rate is basis points and the allowance is an integer floor division,",
    "so two implementations of the same rule land on the same cent rather than within a cent of",
    "each other. `split` and `unapplied_cents` are scored on EXACT equality; a kit that scored",
    "them to the nearest dollar could not see the defect this rule exists to prevent.",
    "",
    "⚑ FOUR THINGS ARE GRADED AND THE SET OF IDS IS ONE OF THEM. Getting the money right against",
    "the wrong invoice is not a partial success: the cash clears, the aged debt report balances,",

The outcomeWhat a good result looks like

A proposed application per payment with its evidence attached -- the invoices, the cents each receives, the cash left unapplied, the receivable left open, and one sentence per invoice saying why. A person accepts or rejects; nothing posts.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• A remittance stream that mostly carries a clean invoice number -- an EDI 820, a portal, a customer whose AP system always writes the reference the same way — the free rules floor, and nothing else
  There is nothing to read, and that is exactly where the floor is perfect: 36 of 36 clean payments on this corpus, in no time at all, with no key and no network. The model scores 31 of the same 36.
• A queue where payers routinely cite purchase orders, statement periods or nothing, and take deductions the terms may or may not permit — the model, on the payments the regex cannot resolve
  This is the whole of what it buys: 5 of 5 on purchase orders, 3 of 4 on statement months and 4 of 4 on discounts taken outside their window, against 0 for the floor on every one of them.
• Anywhere the cost of applying cash to the wrong invoice is high — either arm, with the four-field scorer and the arithmetic in code
  No arm here ever misapplied cash -- 0 of 9 payments the key leaves alone, on every arm. That is a property of the conservative selection rule (two invoices matching equally means apply nothing), not of the model.

At a glanceHow the whole thing runs

Run once, for real, on 2026-09-10. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-09-10 — r001-cash-application. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Observed on the committed kit this session, with the model key blanked for the process: API_KEY= python3 -m src.app starts the board on its assigned port and every panel renders -- the payment, the ledger, the answer key, both free floors and the whole arithmetic station -- with the model button saying plainly that no key is configured and nothing was called. python3 tools/build_corpus.py --check, python3 -m evals.check_labels, both floors and python3 -m evals.router all completed with no network. requirements.txt names nothing: there is no install step, because the kit is the Python standard library and nothing else.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one payment -- one application, one model call, end to end with reasoning explicitly DISABLED on the wire, over a shared connection. The arithmetic station adds nothing measurable: it is pure Python over the reply. ⚠︎ NOT AN SLA. p50 1359 ms and p95 1890 ms across 60 calls; the run finished in 14.5 s of wall clock at 5 workers. The free rules floor answers all 60 payments with no network at all, the corpus generator rebuilds and byte-compares the whole corpus, and the label gate re-grades the answer key with independent arithmetic in 0.06 s -- all measured this session on the committed kit.

Current processWhat it replaces

The read-the-remittance-beside-the-ledger step of cash application: deciding which open invoices one incoming payment settles and how the cash splits across them.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one applications -- one per payment is

No train/test split, because nothing is trained or tuned, and no segmentation, because the payment and its whole visible ledger go into one call. The unit every rate stands on is the PAYMENT: 60 payments, 60 applications, 240 graded checks. 24 are HARD and 36 CLEAN, fixed by the generator before any arm answered -- hard means the note has to be READ, not pattern-matched. Fifteen planted case families with the count published per family. ⚑ THE CLEAN/HARD SPLIT IS THE WHOLE FINDING: the free rules floor is 36 of 36 on the clean payments and 11 of 24 on the hard ones; the model is 31 of 36 and 20 of 24.

SetupWhat the setup figure measured

There is no index to build and no retrieval step: the payment and its visible ledger go whole into one prompt at 2595 input tokens on average. The 0.06 s is the LABEL GATE re-deriving the whole answer key with independent arithmetic and re-checking every published count. Beside it, the corpus generator rebuilds all 60 payments, 357 invoices and the 240-check key and proves it byte-identical to what is committed under three different PYTHONHASHSEEDs, and each free floor scores all 60 payments including writing its result file. That is what the entire free half of this kit costs to reproduce: under a second, and no network.

LicenceLicence

MIT -- and the MIT text is in the repository's LICENSE-PUBLIC file, NOT in its root LICENSE, which governs the repository itself and grants a reader nothing. The KIT and its corpus are MIT; the repository is not, and this line says which, because it is the only claim a reader is entitled to rely on before forking. There is no third-party data in it to licence at all: the payments, the ledger, the rulebook and the answer key are generated in-process from a fixed seed. data/SOURCES.md states the same distinction.

Bring your ownBring your own payments, each with the open ledger put in front of it

Replace data/invoices.json and data/payments.json with your own rows in the same shape -- invoice_id, customer_id, issue_date, due_date, amount_cents, po_number, terms_discount_bps, terms_discount_days, credit_note_cents, status; and payment_id, customer_id, payer_name, value_date, amount_cents, remittance_note. Every amount is an INTEGER NUMBER OF CENTS and the kit will refuse a float. Then edit data/fields.json so the rulebook matches your own trade terms -- that one file is read by the prompt, the station, both floors, the scorer and the generator, so a term you change moves all five at once. Write your own gold.jsonl in the same shape and python3 -m evals.check_labels will grade it before you spend anything: it will tell you if a key row grants an allowance your own rulebook forbids.

⚠︎ And what stops being true when you do: ⚠︎ WHAT STOPS BEING TRUE the moment you swap the corpus: every accuracy figure on this page, and the comparison between the arms. The finding here is that a free regex floor and a paid model are within noise of each other ON THIS MIX -- 36 clean payments where the floor is perfect and 24 hard ones where it is not. Shift the mix towards remittances that carry a clean invoice number and the floor wins outright; shift it towards purchase orders, statement periods and disputed deductions and the model does. The kit's transferable part is the METHOD -- four graded fields, a free floor run first, the arithmetic in code -- not the percentages.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 2,624 tokens assembled, 1,021 are instructions — 39% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The assembled prompt for the PMT-0019 call shown here, replayed from the kit's own src/prompt.py build(). That is only trustworthy because the replayed part sizes and the section list match what the run itself recorded: the run file carries prompt_parts [2934, 3514, 410, 3233, 1267] and sections_used ['system', 'rulebook', 'payment', 'ledger', 'schema'], and the replay produces the same five parts in the same order. A reader cannot check that unless the page says it.

Full promptThe prompt in full

You are applying ONE incoming customer payment to the open invoices it was meant to
pay. Your output is what a cash-application analyst works from: which invoices the payment
settles, how much of it each one receives, what is left unapplied, and the evidence for all of it.

⚠︎ YOU PROPOSE. YOU DO NOT POST. Nothing you return moves money or closes an invoice; a person
accepts or rejects it. That is why leaving a payment unapplied is a real answer and not a
failure -- an unapplied payment sits on a report somebody works, and a payment applied to the
wrong invoice does not.

How to work:

- THE LEDGER IS THE AUTHORITY ON WHAT IS OWED. The remittance note is what the payer says. Where
  they disagree, the ledger is right: a payer naming an invoice number does not make that invoice
  theirs to pay, does not reopen one that was settled, and does not change what it is for.
- READ THE NOTE FIRST, AND READ ALL OF IT. Payers write references however they like -- with the
  hyphen, without it, in lower case, or as the five digits alone -- and they do not always give an
  invoice number at all. Three kinds of reference resolve, and this corpus's payers use all three:
  an INVOICE NUMBER names that invoice; a PURCHASE ORDER NUMBER names every open invoice raised
  under it; a STATEMENT PERIOD -- a month and a year in words -- names every open invoice issued
  in that month. A reference is a strong hint and it is not a decision: it can name an invoice
  that is not in the ledger, one already settled, or one held for a different customer entirely.
- THEN JUDGE THE LEDGER ROW BY ROW. An invoice can receive money only if ALL THREE tests pass:
      customer   it is held for the customer that sent the payment
      status     it is OPEN -- a settled invoice cannot be paid twice
      issued     it was issued on or before the payment's value date
- WHEN THE NOTE RESOLVES TO ELIGIBLE INVOICES, THOSE ARE THE ANSWER, and nothing else is
  considered. When it resolves to nothing eligible, fall back to the amount -- and only when the
  amount fits exactly ONE eligible invoice. When two fit equally, apply nothing.
- SAY WHY THE CASH IS SHORT OF AN INVOICE, using the four words below and no others. A settlement
  discount is a discount ONLY inside the invoice's own stated window, counted from the ISSUE date;
  outside it, the same money is a short pay and the balance stays open. A credit note allows only
  what the invoice itself carries -- a credit note number in a note allows nothing on its own.
- IF NOTHING CAN BE APPLIED, SAY SO. The invoice list is empty, the whole payment is unapplied,
  and that is a complete answer. Do not pick the closest invoice.

Every amount you write is an INTEGER NUMBER OF CENTS. Not dollars, not a decimal, not a string:
$18,400.50 is 1840050. What you apply plus what is unapplied must equal the payment exactly.

Reply with JSON and nothing else, in the shape given at the end.

THE RULEBOOK this cash application is performed under.

Eligibility -- an invoice can receive money only if every test passes:
  customer   the invoice is held for the customer that sent the payment. A payer citing an invoice number does not make somebody else's invoice theirs to pay.
  status     the invoice is OPEN. A settled invoice cannot receive money again, however plainly the remittance note names it -- that is a duplicate application, and it is the error that takes a month to find.
  issued     the invoice was issued on or before the payment's value date. Money cannot arrive for an invoice that had not been raised.

Selection -- which invoices the payment is against:
  1. READ THE NOTE FIRST. When the remittance advice names anything that resolves to eligible invoices, those invoices are the set -- nothing else is considered. Three vocabularies resolve, and customers use all three: an INVOICE NUMBER resolves to that invoice; a PURCHASE ORDER NUMBER resolves to every eligible invoice carrying that PO; a STATEMENT PERIOD (a calendar month) resolves to every eligible invoice ISSUED in that month for this customer.
  2. OTHERWISE FALL BACK TO THE AMOUNT, AND ONLY WHEN IT IS UNIQUE. When the note names nothing that resolves, the payment is applied to an eligible invoice only if its amount equals the NET OWED on exactly ONE of them. Net owed is the invoice amount less any allowance the rulebook below permits.
  3. OTHERWISE APPLY NOTHING. The set is empty and the whole payment stays unapplied on account. Two invoices that both match the amount is rule 3, not a coin toss: an ambiguous payment posted to the wrong account is worse than an unapplied one, because the unapplied one is on a report somebody works.

Allowance -- why the cash may be less than the invoice. One word per invoice:
  full         nothing is deducted; the invoice is paid in full. allowance = 0.
  discount     the early-payment discount the invoice's own terms allow, and ONLY when the payment's value date is within terms_discount_days of the invoice ISSUE date, inclusive. allowance = terms_discount_bps * amount_cents // 10000, INTEGER FLOOR DIVISION. A discount taken outside the window is not a discount -- it is a short pay, and calling it a discount writes off money the customer still owes.
  credit_note  an approved credit note already stands against this invoice. allowance = the invoice's own credit_note_cents, exactly. A credit note that does not exist on the invoice allows nothing.
  short        no allowance applies. The invoice receives whatever cash reaches it and the rest stays open, to be chased. allowance = 0, and the shortfall is a receivable, not a write-off.

Draw order:
  1. Once the set is chosen, the invoices are drawn OLDEST ISSUE DATE FIRST.
  2. Ties on issue date are broken by invoice_id ascending, so two implementations of this rulebook agree on every cent.
  3. Each invoice receives min(what is left of the payment, amount_cents - allowance_cents for its reason).
  4. Whatever is left of the payment when the set is exhausted is unapplied_cents. It is never negative: a payment cannot apply more cash than arrived.

The verdict is DERIVED from the amounts and is never decided separately: applied when unapplied_cents is 0 and no invoice in the set is left short; on_account when nothing was applied at all; partial otherwise. It is DERIVED from the amounts and never decided separately -- a word that can disagree with the numbers beside it is a word nobody can trust.

THE PAYMENT -- cash that has already arrived in the bank.

  payment id        PMT-0019
  payer on the bank statement   Pinegate Supply Co (AP)
  resolved account  Pinegate Supply Co (CUST-061)
  value date        2026-08-15   <- the date the `issued` and discount-window tests use
  AMOUNT RECEIVED   1732775 cents  (USD $17,327.75)

  the remittance advice, exactly as the payer sent it:
    PO-46945 settlement, no invoice breakdown available from our AP system.

THE LEDGER put in front of you, oldest invoice first.
It holds every OPEN invoice for this customer, PLUS any invoice the remittance note names even when it is settled or held for someone else -- so that you can turn it down.

  INV-10350   Seat expansion
    billed to           Pinegate Supply Co (CUST-061)
    status              OPEN
    issued / due        2026-02-08 / 2026-03-25
    purchase order      (none)
    invoice amount      2328300 cents  ($23,283.00)
    settlement terms    2.00% within 10 days of issue -> 46566 cents ($465.66); this payment landed 188 day(s) after issue, so the window is CLOSED
    credit notes        none on this invoice

  INV-10352   Training credits
    billed to           Pinegate Supply Co (CUST-061)
    status              OPEN
    issued / due        2026-02-09 / 2026-03-26
    purchase order      PO-46945
    invoice amount      1279800 cents  ($12,798.00)
    settlement terms    1.50% within 15 days of issue -> 19197 cents ($191.97); this payment landed 187 day(s) after issue, so the window is CLOSED
    credit notes        none on this invoice

  INV-10351   Seat expansion
    billed to           Pinegate Supply Co (CUST-061)
    status              OPEN
    issued / due        2026-02-11 / 2026-03-28
    purchase order      PO-43046
    invoice amount      194300 cents  ($1,943.00)
    settlement terms    none -- no early-payment discount on this invoice
    credit notes        none on this invoice

  INV-10349   Platform subscription
    billed to           Pinegate Supply Co (CUST-061)
    status              OPEN
    issued / due        2026-02-25 / 2026-04-11
    purchase order      PO-46945
    invoice amount      452975 cents  ($4,529.75)
    settlement terms    none -- no early-payment discount on this invoice
    credit notes        none on this invoice

  INV-10353   Premium support
    billed to           Pinegate Supply Co (CUST-061)
    status              OPEN
    issued / due        2026-06-22 / 2026-08-21
    purchase order      (none)
    invoice amount      7718700 cents  ($77,187.00)
    settlement terms    none -- no early-payment discount on this invoice
    credit notes        none on this invoice

  INV-10348   Sandbox environment
    billed to           Pinegate Supply Co (CUST-061)
    status              OPEN
    issued / due        2026-07-28 / 2026-09-11
    purchase order      PO-46973
    invoice amount      2692600 cents  ($26,926.00)
    settlement terms    2.00% within 10 days of issue -> 53852 cents ($538.52); this payment landed 18 day(s) after issue, so the window is CLOSED
    credit notes        none on this invoice

Reply with JSON and nothing else, exactly this shape:

{"cited_text": "<the part of the remittance note that names something, quoted verbatim from it;
                 null when the note names nothing>",
 "cited_resolves_to": ["<every invoice id in the ledger above that the note names, however the
                        payer spelled it -- INCLUDING ones that turn out to be settled or held for
                        another customer. This is what you READ, not what you decided.>"],
 "applications": [
   {"invoice_id": "<an eligible invoice id from the ledger above>",
    "reason": "full" | "discount" | "credit_note" | "short",
    "applied_cents": <integer cents this invoice receives>,
    "why": "<one sentence: why this invoice, and why that reason>"}
 ],
 "unapplied_cents": <integer>,
 "verdict": "applied" | "partial" | "on_account"}

`cited_resolves_to` is an empty list when the note names nothing that appears in the ledger.
`applications` holds only invoices you are actually applying cash to, and is an empty list when
nothing can be applied -- then `unapplied_cents` is the whole payment and `verdict` is
"on_account". The `applied_cents` figures plus `unapplied_cents` must equal the payment exactly.
Every amount is an integer number of cents.

Raw responseThe raw response

{
 "cited_text": "inv-10001",
 "cited_resolves_to": [
  "INV-10001"
 ],
 "applications": [],
 "unapplied_cents": 1068300,
 "verdict": "on_account"
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

The invoices one incoming payment was meant to pay — 60 incoming payments drawn from 60 real payments, each with the open ledger put in front of it. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Every grader is pure code and no model judges anything, which is why re-scoring costs $0.00 and returns the same answer every time. The four graded fields are compared with integer == against a key that was itself graded, before any call was bought, by an independently written third derivation of the same rulebook.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar in cost_by_model is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anybody paid. The token counts come from the provider's own usage block on each call. ⚑ AND NO OUTPUT TOKEN HERE IS REASONING: the budget was sent disabled, so unlike most kits in this estate the projected output side is the answer and nothing else.

Rates checked 2026-09-05. The provider that actually ran every call here is kept off this page per the series rule. Its real spend, its own dated card, its peak/off-peak split and its cached-input tier are recorded per call in results/eval-r001-cash-application.json and summarised in Cost.measured_on, not projected here.

Grading unitWhat the grading figure prices

Free means free: the graders reach no provider, so one eval pass over a cached run costs nothing and can be repeated as often as you like. ⚠︎ THAT IS THE COST OF THE RULER, NOT OF THE THING IT MEASURES -- the 60 calls the run itself bought cost $0.0248, and they are priced in the Cost lens.

The gradersThree ways to grade

⚑ THE FLOOR IS NOT A STRAW MAN AND THE KIT WOULD BE DISHONEST IF IT WERE. Its regex accepts INV-10412, INV10412, inv 10412 and lower case -- every spelling in the corpus -- and it does every piece of arithmetic in the job correctly, because it hands its judgement to the same station the model's answer goes through. It is perfect on all 36 clean payments. What it cannot do is read a purchase order or a statement month as a set of invoices, or tell a settlement discount taken on time from the identical number taken six weeks late -- and those are the only places the paid arm earns anything.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — top_k, chunk size, prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-09-05. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingGrading is free; the bill is the 60 calls that answered.

⚠︎ THAT IS THE RULER, NOT THE THING MEASURED. The run itself bought 60 calls for $0.024758. A forker re-running the pipeline pays that again; re-running the EVAL over what they already bought is free.

Cost driversWhat actually moves the bill

• THE RULEBOOK, not the payment. The role and the rulebook are 60 pct of the 2624 input tokens on the call shown, and they are byte-identical on every call in the run -- which is why 45.2 pct of the run's input came back as prefix cache hits and why a provider with a cached-input tier changes this bill more than a cheaper headline rate would.
• THE SIZE OF THE PAYER'S OPEN LEDGER. It goes in whole -- 5.55 invoices on average here -- with no blocking step and no shortlist. A payer with ten times the open balance is ten times the ledger block.
• REASONING, by its absence. It is sent OFF explicitly and the run reports zero reasoning tokens. Sibling kits on this tier that left the provider default in place recorded 83 pct of their output as reasoning nobody sees; that single field is the largest cost lever in the kit and it is already pulled.
• The number of invoices APPLIED, which sets the reply length -- one object each. Output averaged 138.7 tokens and is the smaller side of this bill.

Your volumeWhat it costs at your volume

NOT LINEAR, and it gets cheaper per payment before it gets dearer. Ten times the payments over the same payer base re-sends the same 60-pct fixed prefix, so the cached-input share rises and the marginal call costs less than the average one. Ten times the LEDGER per payer is the opposite and it is the real cliff: the ledger block grows linearly with a payer's open balance, with no index and no shortlist to absorb it.

Model choiceWhy this model

One provider, one key, one model -- the rule for every kit in this series, because a forker holds one credential and a bake-off is a different kit. The tier chosen is the cheapest long-context option this estate carries, and reasoning was disabled explicitly, which is what makes $0.0248 buy 60 calls.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

the whole 60-payment scored run (r001-cash-application, the fast tier): 155723 input tokens, 70400 of them prefix cache hits, and 8324 output of which NONE is reasoning -- the budget was sent disabled. All 60 calls answered, 0 failures, so this total covers every call in the denominator.

The codeEvery file, and what it is for

10 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pythe corpus generator

Writes the ledger, the payments and the answer key from one seeded random.Random(20260910). ⚑ IT REFUSES TO WRITE AT ALL unless the rulebook's own selection rule, applied to the finished corpus as an arm would apply it, reproduces what it constructed -- a key only its author can reproduce is an opinion. That check caught 8 wrong rows on the first build and 3 on the second, both times a case that edited the ledger invalidating an earlier payment's key; the fix was one payer per payment. --check rebuilds and byte-compares.

# Write the ledger, the payments and the answer key. Deterministic, offline, no key, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
SEED = 20260910
DATASET_VERSION = "cash-application-v1-60payments"
CASE_MIX = [
HARD_CASES = {"lump_sum_po", "lump_sum_statement", "discount_not_earned", "credit_note_absent",
FIRST = ["Northwind", "Blue Harbor", "Cedarline", "Ironbridge", "Meridian", "Kestrel", "Alder",
LAST = ["Analytics", "Logistics", "Systems", "Partners", "Works", "Group", "Labs", "Holdings",
PRODUCTS = ["Platform subscription", "Seat expansion", "Premium support", "Data egress",

src/ledger.pythe visible ledger — a swap seam

The ONLY reduction step in this kit, pure code with no model in it. It cuts 357 invoices down to the ones held for THIS payer and still open, then adds any invoice the note names -- even a settled one, even another customer's -- so the arm can turn it down, and sorts the result oldest issue date first, which is also the rulebook's draw order. Nothing is indexed and nothing persists.

You change it to: visible() -- the single rule about which ledger rows go in front of one payment. Widen it and the prompt grows; narrow it and a miscited invoice becomes invisible and therefore un-rejectable.

# Loading the corpus, and the ONE rule about what a cash application is allowed to see.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
SPEC = json.load(open(os.path.join(DATA, "fields.json"), encoding="utf-8"))
INVOICE_RE = re.compile(SPEC["invoice_pattern"])
PO_RE = re.compile(SPEC["po_pattern"])
MONTHS = {m: i + 1 for i, m in enumerate(
def invoices():
def payments():
def gold():

src/prompt.pyprompt assembly

Five parts in a fixed order -- the role, the rulebook rendered from data/fields.json, the payment with the remittance advice verbatim, the ledger with every allowance and discount window already computed, and the JSON shape. It states the whole rulebook because none of it is what is being measured, and it prints the discount already floored rather than making the arm re-derive a floor division from a percentage. It declares the verdict vocabulary once and asserts at import that every verdict appears in the schema the model is sent.

# The assembled prompt, in one place, in the order it is sent.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
SPEC = json.load(open(os.path.join(HERE, "data", "fields.json"), encoding="utf-8"))
GRADED = SPEC["graded"]
VERDICTS = ["applied", "partial", "on_account"]
VERDICT_MEANINGS = {
SYSTEM = """You are applying ONE incoming customer payment to the open invoices it was meant to
def rulebook_block():
def payment_block(p):
def ledger_block(p, visible):

src/matcher.pythe model call

One call per payment, and the only place a model is reached. Reasoning is sent OFF explicitly (THINKING_OFF), the ceiling is 4000 tokens, and the largest reply in the run drew 305 of them (7.6 pct). Parses a reply that may arrive fenced or with prose in front of it by walking the braces rather than regexing them, and returns the RAW answer and the RECHECKED one side by side. A reply cut off at the ceiling is recorded with at_ceiling set and stays inside the published denominator; it is never re-fired.

# One payment in, one proposed application out. The only place a model is called.

MAX_TOKENS = 4000
THINKING = THINKING_OFF
def parse_reply(text):
def match(cfg, p, visible, complete_fn=None, max_tokens=None):

src/apply.pythe arithmetic station

Pure code, and the reason the money is checkable. Takes a judgement -- which invoices, and one allowance word each -- and produces every cent: the draw order, the allowance, the subtraction, all in integer cents with no tolerance and no configuration. It is a RECHECK, not a repair: given the wrong invoices it computes exact amounts against the wrong invoices, as it should. It also derives left_open_cents from what the arm itself said, so the raw column measures the arm rather than the station.

# The arithmetic station. Given a judgement -- which invoices, and why the cash is short of each

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
SPEC = json.load(open(os.path.join(HERE, "data", "fields.json"), encoding="utf-8"))
GRADED = SPEC["graded"]
GRADED_NAMES = [g["name"] for g in GRADED]
VERDICTS = ["applied", "partial", "on_account"]
REASONS = ["full", "discount", "credit_note", "short"]
def _int_cents(v):
def verdict_of(unapplied_cents, left_open_cents, applied_total_cents):
def eligible(inv, pmt):

evals/baseline.pythe free floors — a swap seam

Two arms, no key and no network. exact matches a face amount and reads nothing; rules pulls invoice numbers out with a regex that accepts every spelling this corpus's payers use, keeps the eligible ones and falls back to the amount. Both hand their judgement to the same station every other arm uses, so a comparison between arms is a comparison of judgements and nobody's arithmetic is an advantage.

You change it to: The number the paid arm has to earn. Strengthen the regex, add PO handling, and the model's remaining margin moves -- which is the honest way to decide whether to buy anything at all.

# The free floors. No key, no network, no model -- and they are the number the paid arm has to

MODES = ("exact", "rules")
INV_RE = re.compile(r"\bINV[-\s]?(\d{5})\b", re.I)
def _eligible(visible, p):
def _candidate_nets(inv):
def _amount_fallback(p, elig, accept_allowances):
def code(p, visible, mode="rules"):
def _byid(visible, iid):
def _reason(p, inv, n_named):

evals/scoring.pythe scorer

Four graded fields per payment, integer == throughout, 240 checks in total. It counts the error DIRECTIONS separately in rows and in money -- cash misapplied, cash left in suspense, receivable written off wrongly -- because thirty rows wrong by a dollar and one row wrong by nine thousand are not the same finding.

# Grade one arm against the answer key. Deterministic -- no model judges anything here.

GRADED_NAMES = AP.GRADED_NAMES
VERDICTS = AP.VERDICTS
TAXONOMY = ("payment_unanswered", "misapplied_cash", "left_in_suspense",
def _pct(n, d):
def citation_read_ok(g, got):
def reason_scores(g, got):
def score(arms, golds, pmts):
def _bucket(g, got, arm_ids, arm_split, want_split, i_ok, s_ok, u_ok, l_ok, lopen):
def _tax(taxonomy, bucket, cents, pid, g, got):

evals/check_labels.pythe label gate

Grades the ANSWER KEY, before any arm is graded against it, with arithmetic written differently on purpose and importing neither the generator nor the station. It is the third independent derivation of the same rulebook.

# Grade the ANSWER KEY, before any arm is graded against it. No key, no network, no model.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
SPEC = json.load(open(os.path.join(DATA, "fields.json"), encoding="utf-8"))
INVOICES = json.load(open(os.path.join(DATA, "invoices.json"), encoding="utf-8"))
PAYMENTS = json.load(open(os.path.join(DATA, "payments.json"), encoding="utf-8"))
STATS = json.load(open(os.path.join(DATA, "corpus-stats.json"), encoding="utf-8"))
GOLD = [json.loads(l) for l in open(os.path.join(DATA, "gold.jsonl"), encoding="utf-8") if l.strip()]
MONTH_LEN = (0, 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31)
MONTH_WORDS = ("january", "february", "march", "april", "may", "june",

evals/router.pythe routed re-scoring

Reads the two committed arms and scores them as one pipeline under a routing rule decidable before any call is bought. Makes NO call and is not a run. It exists because the obvious conclusion from the per-case table is a router, and the measurement says the obvious router does not work.

# Score the two recorded arms as ONE pipeline: regex first, model only for what it cannot read.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
FLOOR_RUN = "b000-cash-application-rules"
MODEL_RUN = "r001-cash-application"
def regex_can_read(p, visible):
def main():

src/app.pythe local board

Standard library HTTP server. Renders every panel with NO key configured; only the propose button calls a provider. ⚑ NO ROUTE RETURNS THE MODEL ID -- that is structural rather than editorial, so a published screenshot cannot show it by accident.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "9369"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-cash-application")
FLOOR_RUNS = {"exact": "b000-cash-application-exact",
ROUTED_RUN = "c000-cash-application-routed"
def _invs():
def _pmts():
def recorded(run_id):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2595 input and 138 output tokens per query at top-k 5, of which about 150 tokens are fixed (system prompt, chat framing, the question itself) and the rest is retrieved context. Change the retrieval depth and the input scales with it. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model and top-k follow the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

⚠︎ THE PROBE IS WRITTEN, PAIRED AND UNFIRED, SO THIS KIT PUBLISHES NO SUPPRESSION RATE. The threat is not hypothetical for this use case: the remittance advice is the PAYER'S free text, typed into their own accounts-payable system, and it reaches the model verbatim -- and the party who wrote it has a direct financial interest in their invoices closing and no balance being chased. It is already a field where people write sentences addressed to a human clerk; a sentence addressed to whatever reads the file next is the same keystrokes. evals/injection.py exists to measure that and has never been bought. What CAN be said without an attack is architectural, and it is said in both directions: the ledger is read from data/invoices.json and never from the model, so a sentence that simply asserts a larger figure cannot change what an invoice is worth or reach the rechecked answer -- but a sentence that persuades the arm a late discount was earned produces a rechecked answer that closes a live receivable and is just as wrong, by a longer route. That second path is the one this kit's fourth graded field exists to see, and it is the one nobody has attacked.

API_KEY is read from the shared <repo>/.env, this kit's own .env, or the real environment, in that order. Both files are gitignored from the first commit; this repo has never held a credential. The key never leaves the machine, is never printed, and src/app.py strips api_key and base_url out of any exception message before it reaches the browser, so a misconfigured key cannot leak into a UI error. src/budget.py counts CALLS against a shared cap before anything is spent and says out loud when no cap is configured rather than refusing to run a fresh fork. Every free path -- the corpus, both floors, the routed re-scoring, the recorded run, the board -- needs no key at all. ⚑ AND NO ROUTE RETURNS THE MODEL ID: that is structural, so a published screenshot cannot show it and no reviewer has to check a template.

The experimentWe have not attacked it yet -- and the hole is already named

An indirect prompt injection needs a field an outside party controls that reaches the prompt. This use case has one by definition: the remittance advice is written by the payer whose invoices are being settled, and it goes to the model verbatim. The probe that would measure it is written, is paired against the scored run's free cache, and has never been bought. Until it is fired, the five gates above are boundaries confirmed by reading the code rather than payloads run through it -- and the one gate that code cannot close is named as such. Both were run for real on 2026-09-10 -- and what was measured is the SCORED run, not an attack. No adversarial call has been made; see redteam.why.

Two of the five boundaries above hold in CODE and were confirmed by reading it and by probing every route on the running board, not by payloads sent through the model. The one that matters most -- whether a sentence can persuade the arm that a late settlement discount was earned, which produces an arithmetically perfect answer that closes a live receivable -- is 0 trials, and writing its shape down here is what stops a green board reading as a clean bill.

The resultThe ledger cannot be rewritten by a remittance advice; the JUDGEMENT can, and nobody has tried. Structurally, an invoice's amount, status, terms and credit notes come from disk and the rechecked answer recomputes every cent from them, so a sentence asserting a number changes nothing. What is unguarded is persuasion: an arm talked into calling a late discount an earned one produces a perfectly arithmetic answer that closes a receivable the business still owns. That path is unmeasured, it is the path this use case's attacker would actually take, and the fourth graded field is the only thing in the kit that could see it.

The probe is ready and paired: evals/injection.py appends one line to the payer's remittance advice, in the field an accounts-payable clerk actually types in -- a sentence asserting that the shortfall is an approved settlement discount and the invoice should be closed in full. PAIRED means the control is the same payment with the line removed, and the control answers are already in the scored run's cache, so only the attacked half would be bought. ⚠︎ IT HAS NOT BEEN. The count above is 0 rather than absent so that nobody reads a missing figure as a passing one, and no suppression rate, no money-conceded figure and no claim of resistance appears anywhere in this kit. ⚑ THE ATTACK THAT MATTERS HERE IS NOT THE OBVIOUS ONE. A sentence that simply asserts a larger number cannot survive src/apply.py, which recomputes every cent from the committed ledger -- so the rechecked answer is structurally immune to it. The sentence to worry about is the one that persuades the arm a LATE discount was earned: that produces a perfectly consistent answer, with the arithmetic done correctly, which closes a receivable the business still owns. The four discount_not_earned payments in this corpus are that exact failure reached by accident, and the free rules floor commits all four -- $2,565.68 of live receivable -- while the model commits none. That is the only evidence in the kit on this path and it is evidence about an accident, not about an attacker.

HonestyWhat this does not prove

• Any suppression rate. 0 adversarial trials fired.
• Any money-conceded figure under attack.
• Whether the four discount_not_earned payments generalise to a deliberate attack. They are the same failure reached by accident, on 4 rows, with nobody trying.
• Behaviour against a remittance advice that is not plain text -- an attachment, an EDI segment, an image. The corpus has none.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never post a cash application, clear an invoice, close a receivable, write off a balance or write to any system. This kit produces the proposal list a PERSON works through, with the ledger, every eligibility test, the discount window, the allowance and the whole draw printed beside every amount, so they can check the arithmetic rather than believe the total. An applied amount a person cannot check is an applied amount they have to believe.

Stated in the prompt the model is sent (“you propose, you do not post”, and “if nothing can be applied, say so”), on the UI as a banner rather than a footnote, in the README, at the top of data/fields.json, and in src/app.py's docstring -- and enforced by there being no such endpoint: the POST routes return proposals and nothing else, and there is no write path to data/invoices.json, to any ledger or to any payment system.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is ABSENCE OF A WRITE PATH plus a prompt rule, not a runtime enforcement layer -- with one real exception: src/apply.py IS enforcement for everything the rulebook decides by arithmetic. It is NOT enforcement for the JUDGEMENT, and the judgement is where every unfixable error in this kit lives.
• ⚠︎ THE STATION CANNOT FIX A MISJUDGEMENT, AND SAYING IT HELD IS NOT SAYING IT WORKS. If an arm calls a late settlement discount an earned one, src/apply.py faithfully takes the allowance off and returns an amount that is wrong for a reason no care with cents can see -- the invoice closes and a live receivable is gone. Measured: on the model arm the station changed 1 answer of 60, so almost nothing exercised it; the evidence that it does anything at all is on the free floor, whose surviving misses it cannot reach either.
• ⚠︎ THE ZERO ON misapplied_cash IS NOT A CAPABILITY CLAIM. Every arm scored 0, including the one that reads nothing. It is a property of the rulebook's third selection rule -- when the evidence does not decide, apply nothing -- and it would survive a much worse model. Read it as evidence about the DESIGN, not about the arm.
• ⚑ AND IT IS NOT A MONEY ARGUMENT FOR THE CALL. The free rules floor gets 47 of 60 with no key and no network, and the paid arm's 51 of 60 is not distinguishable from it at p = 0.50. The guardrails below are reasons to trust the SHAPE of the answer, not reasons to buy one.

WatchedWhat is watched, and why that one

The kit measures 55 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

19 measured by the latest run 36 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (clean_payments 36, dataset_version cash-application-v1-60payments, failures 0, hard_payments 24, max_tokens 4000, payments 60, payments_answered 60, reason_judgements 71, thinking {"type": "disabled"}) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

match · no model in the path — a baseline, not a peer column — 2 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 2 runs measured the same system — they differ on floor, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

match · with the model in the path — 2 runs. Columns here are only ever compared with each other.

not a time series No two of these 2 runs measured the same system — they differ on calls_bought, dataset_version, max_tokens, output_tokens_max, provider, rescoring_of, thinking, usd_total, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The corpus generator, the label gate, the arithmetic station, both free floors and the routed re-scoring are free and re-run in under a second with no network -- verified this session on the committed kit. Only the model arm costs money: 60 calls, once, $0.024758, fired as an 8-call canary and then --resume for the rest under ONE run id so nothing was bought twice, recorded per call in the run record and never re-fired. Either arm can be re-scored from the cache at any time for nothing. The adversarial arm has never been run.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, standard library end to end -- no orchestration layer, no vendor SDK, no retrieval stack, no numeric or dataframe library, no pip install. requirements.txt names nothing and spends its whole body saying why, including the reason a decimal library is refused: every amount here is an integer number of cents from the generator to the key to the prompt to the scorer, and an integer cannot be 1790.9999999999998, so there is nothing for a Decimal to protect -- while decimal would make ROUNDING configurable, which is exactly the freedom that lets two implementations of one rulebook disagree at a half-cent and both look right. evals/check_labels.py fails on a single float anywhere in the corpus or the key.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear, and concurrent only at the payment level. There is no agent, no tool loop, no retrieval and NO STATE carried between payments -- no invoice is ever marked settled, which is why 5 workers is the entire concurrency story, why the run finished in 14.5 s of wall clock, and why any payment can be re-scored from the cache in isolation. It is also the money-movement boundary expressed as an architecture: a pipeline that carried state would be one that posts.

The other sideWhat a framework costs you

• Adding a provider means editing the PROVIDERS dict by hand. In exchange pip install pulls nothing and a forker runs this on whichever key they already hold.
• The reply is parsed by hand. It held across all 60 paid calls with 0 failures; that is an observation, not a guarantee -- a stricter schema library would fail loudly where this fails quietly.
• No dataframe, so the ledger is a list and the draw is a loop. 357 invoices is nothing; the ceiling this design actually has is the ledger PER PAYMENT, and it is stated in Architecture.breaks_at_scale rather than hidden behind a library that would defer it.
• No rules engine, so the rulebook is one JSON file read by five modules and a change is a data change. That is the point -- the free floor is meant to be what a receivables team would script in a weekend, and the rulebook is meant to be arguable rather than embedded.
• No statistics library, so the paired test that produced this kit's headline (McNemar exact, p = 0.50) is math.comb in twelve lines. It is written out because the CONCLUSION rests on it: a kit that reported 51 against 47 without it would be publishing noise as a result.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-cash-application on the fast tier, 2026-09-10. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 0 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

No band column. None of these readings has an alarm band derived for it yet, so there is nothing for the guardrail board to judge them against. Measured and unjudged is a real state and it is stated once; a column of “no band” is not a band.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 2 runs on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — blocking is computed in memory each run and is the only reduction step.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, FOUNDRY_RUN_ID, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- and the MIT text is in the repository's LICENSE-PUBLIC file, NOT in its root LICENSE, which governs the repository itself and grants a reader nothing. The KIT and its corpus are MIT; the repository is not, and this line says which, because it is the only claim a reader is entitled to rely on before forking. There is no third-party data in it to licence at all: the payments, the ledger, the rulebook and the answer key are generated in-process from a fixed seed. data/SOURCES.md states the same distinction. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe whole application -- the SET of invoices, the per-invoice split, the unapplied cash and the receivable still owed, all four right on one payment

whether each of the 60 proposals is one a cash-application analyst could accept as written. All four, never three: getting the money right against the wrong invoice is not a partial success, because the cash clears, the aged-debt report balances, and two customer accounts are now wrong in opposite directions. It is a wrong answer that RECONCILES, which is the most durable kind and the one nothing downstream ever questions.

MethodHow the test was run

python3 -m evals.run --run-id <id> [--floor exact|rules]; evals/scoring.py compares a set of ids, a set of (id, cents) pairs and two integers against data/gold.jsonl. No model is in the grading path, and re-scoring a cached run costs nothing.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

the note names an invoice held for a different customer; the amount matches exactly one of the payer's own, so that is what it settles

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl -- 60 rows, one per payment, each carrying the case family, the hard flag, what the note resolves to, the visible ledger, the eligible ids, the allocations with an allowance reason each, all four graded amounts and the derived verdict. The key is COMPUTED from the same rulebook in data/fields.json that the prompt states, then re-derived independently by evals/check_labels.py, and the GENERATOR refuses to write unless the rulebook's own selection rule reproduces it. That is the honest description and it is also the key's own limit: it agrees with the rulebook by construction, so what it proves is internal consistency to the cent, not that this rulebook is what any given company's payment terms actually say.

No true/false rates for this grader. It records 7 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? There is no threshold and nothing is tunable. Every comparison is integer == against the key; an arm is right on a field or it is not. The only judgement call in the whole grader is which four fields are graded, and that list lives in data/fields.json rather than in the scorer.

Cadence: Once per run. Re-scoring a cached run is free and deterministic, so a change to src/apply.py can be re-graded against the same bought calls with --resume --rescore and no provider in the loop.

The decisionWhen to reach for it

In one lineThe error DIRECTIONS, in rows and in money -- cash misapplied, cash left in suspense, and receivable written off that is still owed

which way an arm is wrong, because the two directions cost completely different things and a single accuracy figure averages them into nothing. Cash misapplied clears against an invoice that did not receive it and the ledger still balances; cash left in suspense is an analyst reading a remittance. And the third is the one this kit was built to see: a receivable an arm silently closed.

MethodHow the test was run

the same run; these come out of the same pass as the grader above.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

the note names an invoice held for a different customer; the amount matches exactly one of the payer's own, so that is what it settles

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the same data/gold.jsonl. The money figures are differences against the key's own split, per invoice, in integer cents.

No true/false rates for this grader. It records 6 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? None. Money is compared in integer cents with ==; there is no tolerance anywhere in this kit, because a tolerance is exactly what would hide a systematic over-application of a few dollars repeated sixty thousand times a year.

Cadence: Once per run, with the grader above.

The decisionWhen to reach for it

In one lineThe READING -- did the arm pull the right references out of the payer's own words, and did it name the right allowance for each invoice

whether a miss is a reading problem or a judgement problem. An arm that never found the reference needs a better prompt; one that found it and then applied the rulebook wrongly needs something else entirely, and the two cost different things to fix.

MethodHow the test was run

the same run; the same pass.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

the note names an invoice held for a different customer; the amount matches exactly one of the payer's own, so that is what it settles

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: gold.cited_resolves_to, which is DERIVED from the finished note by the same src/ledger.py::named_in_note() every arm and both floors use -- never asserted by the generator. The generator asserts only that what it built the note to name is a subset of what reads back out of it, so the reading grade can never be measured against an intention the note does not carry.

No true/false rates for this grader. It records 5 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? None. Set equality on the resolved reference; string equality on the allowance word against a closed four-word vocabulary declared in data/fields.json.

Cadence: Once per run.

The decisionWhen to reach for it