The business caseThe problem this solves

Reconciling a vendor's payment-status inquiry means reading the invoice's full match/approval/run-inclusion/remittance trail, working out which of the four stages is the real bottleneck -- never trusting whatever a downstream field happens to show -- and drafting a reply grounded in that stage, flagged for AP review whenever an exception is still open, for every inquiry, every invoice. AP staff reading a vendor's payment-status inquiry against its full match/approval/run-inclusion/remittance trail by hand, working out which of the four stages actually governs before drafting a reply -- for every inquiry, every invoice.

The inputThe actual invoices

The corpus is 35 invoices, 0.02 MB (jsonl 1). A real vendor payment run names a real vendor, a real bank reference and a real approval chain -- this repo has never held one and is not able to. There is no public corpus of paired (invoice 4-stage trace, vendor inquiry, adjudicated current stage) records, for the same reason no public corpus of production alerts or supplier agreements exists: the interesting cases are exactly the ones nobody can publish. Nothing in the corpus refers to a real company -- vendor names, invoice numbers, PO numbers and amounts are all invented.

{"invoice_id": "INV-2024-0600", "vendor_name": "Larkspur Catering Group", "po_number": "PO-31257", "amount": 49904.91, "submitted_date": "2024-07-02", "inquiry": "Following up on INV-2024-0600 -- any update on payment timing?", "match": {"matched": true, "reason": null}, "approval": {"status": "exception", "reason": "amount exceeds manager approval threshold, escalated"}, "run_inclusion": {"included": false, "run_id": null, "scheduled_date": null, "reason": "not yet eligible for run inclusion; approval not complete"}, "remittance": {"remitted": false, "remittance_date": null, "method": null, "reference": null, "reason": "not yet remitted; approval not complete"}}
{"invoice_id": "INV-2024-0853", "vendor_name": "Northgate Logistics", "po_number": "PO-16783", "amount": 30450.06, "submitted_date": "2024-06-07", "inquiry": "Following up on INV-2024-0853 -- any update on payment timing?", "match": {"matched": true, "reason": null}, "approval": {"status": "exception", "reason": "invoice amount exceeds the original purchase order by more than tolerance, escalated for review"}, "run_inclusion": {"included": false, "run_id": null, "scheduled_date": null, "reason": "not yet eligible for run inclusion; approval not complete"}, "remittance": {"remitted": false, "remittance_date": null, "method": null, "reference": null, "reason": "not yet remitted; approval not complete"}}

The outcomeWhat a good result looks like

A traced current stage, an AP-review flag derived from it, the one real date (if any) the record supports, and a drafted reply grounded in that stage -- informational only, never releasing a payment or committing to a date beyond the invoice's own scheduled run.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Tracing a vendor's payment-status inquiry against its full match/approval/run-inclusion/remittance trail and drafting a grounded reply — the fast tier, over the free rule-order floor
  100% stage accuracy and 0% false_paid this run, against the floor's 82.9% and 100% false_paid rate on the exact invoices built to be misread downstream-first.
• Deciding whether a reply can state a payment has actually gone out — the fast tier's current_stage verdict, read alongside whether the reply text itself makes a paid claim
  0 of 4 gold remittance-trap invoices this run were told to a vendor as paid (false_paid=0) -- the expensive-direction error this kit's guardrail exists to catch.
• What this kit is not — an informational status reply for AP to review before it reaches a vendor -- never an approval or a payment release
  src/payrun.py and src/app.py have no function anywhere that releases a payment or changes a remittance detail; every reply with an open exception is flagged requires_ap_review before it goes out.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-20. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-20 — r001-fin-payrun. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with no key configured renders the whole panel -- every invoice's full 4-stage record and its vendor inquiry -- since data/invoices.jsonl and data/gold.jsonl are committed, not fetched (python -m src.app). Clicking Trace with no API_KEY returns a calm 200 explaining nothing was called (src/app.py's /api/check). It cannot reproduce a live trace, a score, or a dollar figure without a key -- those are what results/eval-r001-fin-payrun.json already committed.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per vendor inquiry: one HTTP request carrying the invoice's full 4-stage record and the vendor's inquiry text, parsed to a traced current stage, an AP-review flag, a stated date and a drafted reply. No retrieval step -- loading an invoice from disk is pure code and costs no network time. This run left provider-side reasoning ('thinking') at its documented default (on) rather than disabling it, so p50/p95 include a hidden reasoning pass on every call, not only the JSON-formatting cost -- see not_good_enough.

Current processWhat it replaces

AP staff reading a vendor's payment-status inquiry against its full match/approval/run-inclusion/remittance trail by hand, working out which of the four stages actually governs before drafting a reply -- for every inquiry, every invoice.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• jsonl
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one characters per invoice's assembled record block is

One invoice, its full 4-stage record and one vendor inquiry included whole -- no train/test split, every invoice traced once, in one call.

SetupWhat the setup figure measured

No index is built. src/app.py finds an invoice by scanning C.invoices() for a matching invoice_id -- 35 rows, a linear scan, not a search structure -- and src/payrun.py sends that invoice's full record whole into the one call. build_seconds and build_cost_usd are both zero because there is no index-build step, not because one ran for free.

LicenceLicence

MIT (this repository's own licence)

Bring your ownBring your own invoices

Point tools/build_corpus.py at your own vendors, amounts and stage reasons, or write your own data/invoices.jsonl in the same shape -- src/payrun.py and src/prompt.py read an invoice by its own fields (match, approval, run_inclusion, remittance, inquiry) and do not care where the values came from. The five-stage taxonomy in src/prompt.py assumes exactly these four gating systems (match, approval, run_inclusion, remittance); a different set of gating stages is a prompt.py change, not a data change.

⚠︎ And what stops being true when you do: The measured 100% stage accuracy and 0% false_paid rate are THIS corpus's inquiry phrasing (a handful of hand-written templates) and THIS corpus's one-trap-family-per-invoice design (TRAP_N=6 of 14 exception-stage invoices, ~43%, split 4 remittance / 2 run_inclusion). A real vendor's own inquiry phrasing, and a real payment system's own trap shapes -- including ones this corpus does not model, like an incomplete rather than unfavourable trail (see breaks_on) -- are both unmeasured by this run.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — but they are the input total apportioned by each part’s share of the characters, not a per-part reading from the provider’s counter. The total is measured; the split is arithmetic, and a part carrying denser text really costs more of it than this says.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 1,120 tokens assembled, 989 are instructions — 88% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Logged verbatim by importing src.prompt and src.payrun directly and calling prompt.build(inv) for INV-2024-0600 (not retyped) -- the literal system and user content src/payrun.py::check() sends. Per-part token counts are a proportional estimate over character share (system 3634 / record 481 chars, 4115 total) applied to this call's own total of 1120 input tokens -- the provider reports only the call's total, matching lenses.LLM.tokens.input exactly, never a per-segment split. The 'record' part covers only the invoice's 4-stage block and inquiry line src/prompt.py::_stage_block() builds -- the surrounding 'Trace the invoice below...' instruction text is part of the user message actually sent (see prompt_verbatim) but is not a separately named segment in the kit's own code.

Full promptThe prompt in full

You trace one vendor invoice through match, approval, payment-run inclusion and remittance, and draft a reply to the vendor's payment-status inquiry grounded in its actual traced state. You are given the invoice's full 4-stage record and the vendor's inquiry text.

THE FOUR STAGES, READ IN THIS ORDER, STOPPING AT THE FIRST ONE THAT IS NOT CLEANLY COMPLETE
  1. match            matched true/false, with a reason if false
  2. approval          status approved/pending/exception, with a reason if exception
  3. run_inclusion     included true/false; if true a run_id + scheduled_date, if false a reason
  4. remittance        remitted true/false; if true a remittance_date + method + reference, if false a reason

The TRUE current stage is whichever of these four is the actual bottleneck, read IN ORDER -- match first, then approval, then run_inclusion, then remittance. Stop at the first stage that is not cleanly complete. This is the trap: a later field can look complete -- a run_inclusion with a real run_id and scheduled_date, a remittance with a real date and reference -- while an EARLIER stage is the one that actually governs. A downstream field that looks done never overrides an earlier stage that is not. Read the whole record before answering; do not stop at the first field that looks reassuring.

THE FIVE VALUES FOR current_stage, EXACTLY ONE APPLIES
  match_exception          match.matched is false -- the invoice has not cleared three-way match and nothing downstream can be trusted regardless of what it shows.
  approval_exception       match is clean, and approval.status is "exception" -- the invoice is held on an approval problem, whatever run_inclusion or remittance happen to contain.
  awaiting_run_inclusion   match and approval are both clean, and run_inclusion.included is false -- approved, but not yet placed on a scheduled payment run.
  in_scheduled_run         match, approval and run_inclusion are all clean, and remittance.remitted is false -- on a scheduled run with a real scheduled_date, not yet paid.
  remitted                 all four stages are clean and remittance.remitted is true -- the payment has actually gone out.

requires_ap_review is true if and only if current_stage is match_exception or approval_exception -- an open exception at either stage means the drafted reply needs AP review before it is sent to the vendor.

THE REPLY MUST BE GROUNDED IN current_stage, NEVER IN A DOWNSTREAM FIELD ALONE
  - Never claim the invoice is paid, or that payment is imminent, unless current_stage is remitted.
  - If current_stage is in_scheduled_run, state ONLY the real scheduled_date from run_inclusion -- never an earlier or invented date, never "accelerated" or "off-cycle" language.
  - If current_stage is remitted, state the real remittance_date and reference.
  - If current_stage is match_exception or approval_exception, the reply must not state a payment timeline at all -- there is nothing to schedule yet -- and must say the item needs AP review.

stated_date is the one date, if any, the reply states: the real scheduled_date for in_scheduled_run, the real remittance_date for remitted, or null for the other three stages -- there is no date to state for an exception or for a stage still awaiting run inclusion.

You never release a payment, never change a remittance detail, and never commit to a date beyond the invoice's own scheduled run. The reply is informational only -- it reports the invoice's actual traced state to the vendor, it never promises to accelerate or move a payment off-cycle, and it is never sent without AP review when requires_ap_review is true.

Trace the invoice below and draft a reply to the vendor's inquiry.

Return a JSON object with exactly four keys: "current_stage" (one of: match_exception, approval_exception, awaiting_run_inclusion, in_scheduled_run, remitted), "requires_ap_review" (true or false), "stated_date" (the one date the reply states, as it appears in the record, or null), "reply" (the drafted reply text, one or two sentences).

INVOICE INV-2024-0600 (vendor Larkspur Catering Group, PO PO-31257, amount $49904.91, submitted 2024-07-02)
1. match: matched=True
2. approval: status=exception  reason: amount exceeds manager approval threshold, escalated
3. run_inclusion: included=False  reason: not yet eligible for run inclusion; approval not complete
4. remittance: remitted=False  reason: not yet remitted; approval not complete
VENDOR INQUIRY: Following up on INV-2024-0600 -- any update on payment timing?

Raw responseThe raw response

{
  "current_stage": "approval_exception",
  "requires_ap_review": true,
  "stated_date": null,
  "reply": "Your invoice INV-2024-0600 is currently held for AP review because it exceeds the manager approval threshold and has been escalated. We cannot provide a payment timeline until the approval exception is resolved."
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Tell a vendor exactly where their invoice is stuck — 35 invoices. One model answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

The grader is evals/scoring.py and it is pure code: exact match on current_stage (the core 5-way classification) and on requires_ap_review (its own field even though it is derived deterministically from current_stage in gold, because the model can get the stage right and the boolean wrong, or the reverse, if it does not apply the derivation itself); false_paid (a keyword scan of the drafted reply plus a current_stage=='remitted' check) counted separately as the expensive-direction error, over exactly the corpus's planted remittance-trap denominator; and date accuracy (does stated_date match the one real date the record supports, or correctly state none). No judge model. The same function scores both evals/baseline.py's free floor and evals/run.py's real run.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran r001 publishes no rate card this repo commits, so nothing here is what was actually paid -- the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page. Reasoning was also left at provider default (on) for this run -- see Business.not_good_enough -- so even the projected figure prices tokens the shipped app's own reasoning-off configuration would not spend.

Grading unitWhat the grading figure prices

Grading calls nothing -- exact match on stage/review/date and a keyword scan for a paid claim in the reply are both pure code. A measured $0.00, not an unpriced one.

The gradersOne way to grade, and why it is the only one

The floor checks remittance FIRST, so any invoice whose remittance field shows remitted:true is called remitted regardless of what match or approval actually say -- by construction it misclassifies all 4 gold remittance-trap invoices as remitted (a 100% false_paid rate, 4 of 4) and the 2 run_inclusion-trap invoices as in_scheduled_run (checked before approval), for 6 total misses -- exactly the corpus's planted trap set (results/eval-b000-ruleorder.json's own misses list). Its overall stage accuracy still reaches 82.9% (29 of 35) because the other 29 invoices have nothing downstream that looks complete while an earlier stage governs, so both orders agree on them. The fast tier's 0% false_paid and 100% stage accuracy on the identical 35 invoices is the real, measured gap a model closes that this floor cannot.

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The rule-order baseline (evals/baseline.py) and the scorer (evals/scoring.py) are both pure code and cost $0.00 to run against either result set. The figure above is this run's own 39,200 input / 9,575 output tokens priced at Google Gemini 3 Flash's published rate -- the same basis cost_per_query_usd uses, not a second, larger spend.

Cost driversWhat actually moves the bill

• The invoice's 4-stage record (match, approval, run_inclusion, remittance) plus the vendor inquiry text -- fixed size per call by construction (input tokens ranged 1,103-1,142 across all 35 calls, essentially flat), the floor every call pays regardless of which stage governs.
• Reasoning ('thinking') was left at the provider's default (on) rather than explicitly disabled for this run -- it consumed 72.7% of the average output-token budget (198.9 of 273.6 tokens/call) and is already priced into cost_per_query_usd, since a provider bills reasoning tokens as completion tokens. src/app.py's live UI hardcodes thinking off, a different, unmeasured configuration -- see Business.not_good_enough.
• The fixed system prompt (3,634 characters -- the stage taxonomy, precedence rule and answer schema) is sent in full on every call, the largest single fixed cost per invoice.

Your volumeWhat it costs at your volume

Linear in invoices: each call is independent and self-contained, with no shared context or retrieval step to amortise. This run's 35 invoices cost about $0.0483 projected onto Google Gemini 3 Flash's published rate, so ten times the set is about $0.483 on the same rate and the same reasoning-on configuration -- arithmetic on the measured per-call rate, not a second run.

Model choiceWhy this model

The fast tier was the only model run against this corpus. No second tier was run -- unlike gap-brief's fast/reasoning comparison -- so this page prices one model, not a trade-off; see Eval.could_not_verify for what a second run would need to answer.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 35 vendor inquiries traced, 35 invoices stage-scored by pure code. This run (r001-fin-payrun) answered 35 of 35 stages and 35 of 35 AP-review flags -- the row this table prices, the same one Cost.cost_by_model[0] uses.

The codeEvery file, and what it is for

Eight modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pycorpus generator — a swap seam

Generates 35 invoice records and vendor inquiries from a fixed seed (SEED=20260819). Gold (current_stage, requires_ap_review, expected_date) is RECOMPUTED from the actual generated 4-stage record by derive_stage(), never from the target_stage the generator aimed for -- --verify independently re-checks every row against the record actually written to disk.

You change it to: Point it at your own vendors, amounts and stage reasons. src/payrun.py and src/prompt.py read an invoice by its own fields (match, approval, run_inclusion, remittance, inquiry) and do not care where the numbers came from.

# Generate the invoice records and vendor inquiries this kit checks against, from a fixed seed.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
SEED = 20260819
STAGE_COUNTS = {
TRAP_N = 6
TRAP_REMITTANCE_N = 4
TRAP_RUN_INCLUSION_N = 2
VENDOR_NAMES = [
MATCH_REASONS = [

src/prompt.pythe prompt — a swap seam

The five-stage taxonomy, the read-in-order precedence rule and the answer schema, declared once and read from here by build() and parse(). States the false-'remitted' failure mode to the model explicitly, as the named failure this task exists to catch, not left for the model to discover.

You change it to: A stage this kit's fixed five-value STAGES tuple does not carry, or a different read order, needs a rewrite of SYSTEM plus REVIEW_STAGES -- a prompt tweak alone will not teach a precedence rule the corpus never plants.

# Assemble the one prompt this kit sends, and parse the one reply it gets back.

STAGES = (
STAGE_MEANINGS = {
REVIEW_STAGES = ("match_exception", "approval_exception")
SYSTEM = (
DEFAULT_PROMPT = "v1"
SYSTEMS = {"v1": SYSTEM}
def _stage_block(inv):
def build(inv, prompt=DEFAULT_PROMPT):
def parse(raw):

src/payrun.pythe AI layer

Loads one invoice by id, calls the model once with its full 4-stage record and the vendor inquiry, parses the reply into a stage, a review flag, a stated date and a drafted reply. The whole AI layer, deliberately short -- the same split fin-invval's invval.py and fin-close's close.py both make. Never releases a payment or changes a remittance detail -- there is no function here or in src/app.py that does either.

# Trace one invoice through match, approval, run inclusion and remittance, and draft a reply to

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
INVOICES = os.path.join(HERE, "data", "invoices.jsonl")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
MAX_TOKENS = 3000
def invoices():
def load_gold():
def check(cfg, inv, complete=None, thinking=None, prompt=P.DEFAULT_PROMPT):

src/adapters/__init__.pythe model adapter — a swap seam

SEAM -- the model. Raw HTTP for every provider (openai-compatible, anthropic), stdlib only. Passes a thinking kwarg through to the provider only when the caller supplies one, and reports token_details.reasoning_tokens when the provider returns it -- the field the reasoning-token finding in Business.not_good_enough is measured from.

You change it to: .env plus the same run again. Any OpenAI-compatible provider or Anthropic.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/budget.pyspend control

An append-only ledger written BEFORE each call, shared by every kit under one .env so the daily call cap is on the KEY rather than per kit. Counts calls, not dollars, on purpose -- see the file's own docstring.

# A cap on live calls, shared by every kit on this machine. No dependency, no service.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
KIT = os.path.basename(HERE)
ROOT = os.path.dirname(os.path.dirname(HERE))
SHARED_ENV = os.path.join(ROOT, ".env")
LEDGER = os.path.join(ROOT if os.path.exists(SHARED_ENV) else HERE, ".calls-ledger.jsonl")
PAID_ARMS = ("r", "x")
FREE_ARMS = ("t", "b")
BOOT = os.urandom(6).hex()
PID = os.getpid()

src/app.pythe app

Static files and two JSON endpoints (/api/invoice, /api/check) on the standard library. /api/check hardcodes thinking=THINKING_OFF on every live call -- a different reasoning setting from the one the registered eval run (r001-fin-payrun) actually used; see Business.not_good_enough.

# The minimal local UI. Standard library only — python -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "8789"))
class H(BaseHTTPRequestHandler):
def main():

evals/scoring.pythe scorer

Stage-level scoring, pure code: exact match on current_stage and requires_ap_review, false_paid (a keyword scan of the reply plus a remitted-stage check) reported as its own named figure rather than folded into a generic 'wrong' bucket, and date accuracy checked separately.

# Score a set of predicted invoice traces against gold. Pure code, shared by evals/baseline.py

STAGES = ("match_exception", "approval_exception", "awaiting_run_inclusion", "in_scheduled_run",
def _reply_claims_paid(reply):
def score(records, gold):

evals/baseline.pyfree baseline

A fixed-priority rule, written the way a person free-texting a quick classifier would write one, that deliberately checks remittance BEFORE match/approval/run_inclusion -- so it misclassifies every planted trap invoice by construction. The honest, narrow floor a model has to clear.

# What a simple, dumb rule catches, over the same corpus. Free. No key, no dependency, no model

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
def classify(inv):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 1120 input and 273 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's vendor inquiries are entirely synthetic (tools/build_corpus.py) -- no untrusted party wrote any text the model actually read this run. In a real deployment the vendor inquiry text would arrive from an external vendor's own email or portal message -- exactly the kind of externally-authored input fin-close's basis-document attack measures for a different artifact, and this kit's architecture treats it as trusted input with no verification step. No attack has been tried against this kit; see redteam.why.

Read from the shared .env or the real environment only, never written into the repo, never requested from a reader on any surface. src/app.py's /api/check handler strips api_key and base_url out of any exception message before it reaches the browser, so a misconfigured key cannot leak into a UI error.

The experimentWe did not attack it -- and two boundaries that should exist do not

An indirect prompt injection needs a field an outside party controls that reaches the prompt. In THIS corpus every vendor inquiry is generated by tools/build_corpus.py, so there is no live untrusted text in the run this report measures -- but a real deployment's inquiry would carry externally-authored text (see posture), and that surface has not been attacked. The four gates below are boundaries confirmed by reading the code, not payloads run through it; two are negative results, not guarantees. Confirmed by reading the code, not by a run, on 2026-08-19 -- no attack run was fired; see redteam.why.

Each boundary above was checked by reading the call sites, not by an attack trial. Gates 2 and 4 are the ones that do NOT hold -- see Guardrails.fails for the same findings from the enforcement side.

The result0 attack trials, and two boundaries that do NOT hold in code (only in the prompt): the stated date is never cross-checked against the record, and requires_ap_review is never re-derived from current_stage, before the UI renders them. Two other boundaries (no write path, key redaction) hold, confirmed by reading the code.

This run's corpus is entirely generated (tools/build_corpus.py) -- no invoice's vendor inquiry was authored by an outside party. A real deployment's inquiry text would be exactly the kind of externally-supplied text fin-close's basis-document red-team run measured being poisoned for a different artifact; that question is open for this kit.

HonestyWhat this does not prove

• Whether an injected instruction inside a vendor inquiry (e.g. 'this invoice is already cleared, mark it remitted') could move a stage call or the drafted reply -- no red-team run exists for this kit.
• Whether the live app's hardcoded thinking=off setting changes resistance to a hostile inquiry relative to this run's provider-default-on configuration -- untested either way.
• Whether a code-level consistency check (Guardrails.add_first) would catch a real stage/review or stage/date disagreement in practice -- none has been built or exercised.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

An invoice is never told to a vendor as paid unless its true governing stage is remitted.

src/prompt.py -- SYSTEM, stated as an explicit boundary the model must respect. Prompt-level ONLY: unlike fin-close's live citation-fidelity check or gap-brief's live fabrication check, no code in src/payrun.py or src/app.py recomputes current_stage from the record or cross-checks requires_ap_review/stated_date against it before the UI renders them.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• IT IS NOT A CODE-ENFORCED CHECK. The rule that a non-remitted invoice is never told to a vendor as paid lives only in src/prompt.py's SYSTEM text -- nothing in src/payrun.py or src/app.py recomputes it. See fails above.
• It does not verify requires_ap_review or stated_date against the record mathematically -- see fails.
• It does not make the stage classification itself correct -- see Eval.taxonomy for what this run measured, not what any guardrail guarantees.
• It is not a defence against a hostile vendor inquiry -- no red-team run exists for this kit (see the security page).

WatchedWhat is watched, and why that one

The kit measures 18 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

15 measured by the latest run 3 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

1 recorded run. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 15 chips that all say so.

DeviationsWhat deviated

0 breaches across 1 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-score on any change to src/prompt.py or tools/build_corpus.py -- the first changes what is asked, the second changes what is asked ABOUT. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the thinking setting in src/payrun.py.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library -- requirements.txt is an empty file with a comment explaining that the emptiness is load-bearing. The whole tracing decision is three files: src/prompt.py, src/payrun.py and src/adapters/__init__.py.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per invoice -- load, assemble, prompt, call, parse -- with no branching and no state carried between invoices. A framework would add an orchestrator to a for-loop.

The other sideWhat a framework costs you

• A gating stage this kit's fixed five-value vocabulary doesn't name needs a hand-written STAGES/STAGE_MEANINGS entry plus new corpus template logic in tools/build_corpus.py, rather than being configured declaratively.
• No built-in retry/backoff beyond src/adapters.py's own bounded retry -- a framework's queue and worker model is not here, so a real deployment adds its own scheduling.
• No built-in observability beyond what evals/run.py prints and writes to results/ -- a framework's tracing/dashboard integration is not here.
• No built-in output validation beyond src/prompt.py::parse()'s own tolerant-but-not-creative JSON parse -- a framework with a schema-and-consistency-check layer baked in might catch a stage/review disagreement before it renders; this kit's stdlib-only design did not build one (see Guardrails.fails).

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-fin-payrun on the fast tier, 2026-08-19. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. This is the only run on record, so there is nothing to move against. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT (this repository's own licence) Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineStage precedence exact match, plus AP-review derivation, false-paid claim scan and date exact match

Does the model's current_stage match gold's precedence-derived stage? Does requires_ap_review match the stage-derived boolean? Does the reply, or the stage itself, falsely claim the invoice is paid when a gold remittance-trap invoice's true stage is still an open exception? Does stated_date match the one real date the record supports, or correctly state none?

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model. The same function evals/baseline.py and evals/run.py both call -- a baseline and a real run scored by two different scorers cannot be compared honestly.

The inputOne real row, seen by every grader

The formulaWhat it computes

stage_accuracy = correct stages / answered invoices, over stage_answered=35. false_paid = current_stage=='remitted' OR reply matches a plain-language paid-claim regex, counted over exactly the gold remittance-trap denominator (4), never folded into stage_accuracy. review_accuracy and date_accuracy are their own exact-match rates.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's derive_stage(), which reads the same 4-stage record IN ORDER and computes the gold current_stage, requires_ap_review and expected_date -- re-derived independently via --verify against the invoice records actually written to disk, asserted zero drift.

How tight can the band be? There is no tolerance band here -- every field is an exact match (stage, boolean, date string) or a plain keyword scan; the AP-tracing task has no continuous quantity to round.

Cadence: Re-run on any change to src/prompt.py, tools/build_corpus.py, or MAX_TOKENS in src/payrun.py -- the first changes what is asked, the second changes what is asked ABOUT, the third bounds how many verdicts can come back at all.

The decisionWhen to reach for it