The business caseThe problem this solves

A receiving discrepancy case can carry 2-4 SKU lines and exactly one is genuinely discrepant -- its PO, BOL and receiving quantities do not all agree. A carrier exception note may or may not be on file, and when one is, it is only evidence for the SKU it actually names, not for the case in general. Reading 'an exception exists somewhere' as 'the carrier did it' produces a liable-party call the documents do not support. Receiving/AP staff reading a receiving-discrepancy case's PO, BOL and warehouse-scan quantities line by line, checking whether any carrier exception note on file actually names the discrepant SKU before proposing who is liable -- for every case, every day.

The inputThe actual cases

The corpus is 36 cases, 0.01 MB (jsonl 1). A real receiving discrepancy names a real vendor, a real carrier and a real chargeback dollar figure -- this repo has never held one and is not able to. There is no public corpus of paired (BOL, receiving scan, carrier exception, adjudicated liable party) records, for the same reason no public corpus of supplier agreements or production alerts exists: the interesting cases are exactly the ones nobody can publish. Nothing in the corpus refers to a real company -- SKUs, PO numbers, case IDs and carrier names are all invented.

{"case_id": "CASE-00001", "po_number": "PO-356479", "carrier_name": "Union Point Freight", "lines": [{"sku": "SKU-4001", "po_qty": 334, "bol_qty": 334, "received_qty": 334}, {"sku": "SKU-4002", "po_qty": 105, "bol_qty": 124, "received_qty": 124}], "carrier_exception": null}
{"case_id": "CASE-00002", "po_number": "PO-193139", "carrier_name": "Union Point Freight", "lines": [{"sku": "SKU-4003", "po_qty": 250, "bol_qty": 250, "received_qty": 250}, {"sku": "SKU-4004", "po_qty": 172, "bol_qty": 172, "received_qty": 132}, {"sku": "SKU-4005", "po_qty": 308, "bol_qty": 308, "received_qty": 308}], "carrier_exception": "1 carton damaged in transit, refused at dock, SKU-4004"}
{"case_id": "CASE-00003", "po_number": "PO-850490", "carrier_name": "Anchor Point Shipping", "lines": [{"sku": "SKU-4006", "po_qty": 440, "bol_qty": 389, "received_qty": 314}, {"sku": "SKU-4007", "po_qty": 435, "bol_qty": 435, "received_qty": 435}, {"sku": "SKU-4008", "po_qty": 196, "bol_qty": 196, "received_qty": 196}], "carrier_exception": null}
{"case_id": "CASE-00004", "po_number": "PO-532927", "carrier_name": "Anchor Point Shipping", "lines": [{"sku": "SKU-4009", "po_qty": 164, "bol_qty": 164, "received_qty": 164}, {"sku": "SKU-4010", "po_qty": 252, "bol_qty": 252, "received_qty": 175}, {"sku": "SKU-4011", "po_qty": 103, "bol_qty": 103, "received_qty": 103}], "carrier_exception": "carrier reported a shortage at pickup for SKU-4010"}

The outcomeWhat a good result looks like

A proposed liable party (vendor, carrier, internal, or insufficient_evidence), the discrepant SKU, the two quantities and delta the notice cites, and a drafted notice -- for receiving/AP to confirm before any chargeback or credit is issued.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Resolving a receiving-discrepancy case's liable party from its PO/BOL/receiving quantities and any carrier exception note — the fast tier, over the free exception-first floor
  100% liable_party accuracy and 0% wrong_sku_exception_misread this run, against the floor's 83.3% and 100% wrong_sku_exception_misread rate on the exact cases built to be misread on exception presence alone.
• Deciding whether a case has a clean documentary basis for a liability call at all — the fast tier's liable_party verdict, read alongside whether it answered insufficient_evidence rather than forcing a guess
  0 of 6 gold insufficient_evidence cases this run were given a confident vendor/carrier/internal call (false_confident_call=0) -- the expensive-direction error this kit's guardrail exists to catch.
• What this kit is not — a proposed liable party and its documentary basis for receiving/AP to review -- never a chargeback, never a credit, never a final determination
  src/resolve.py and src/app.py have no function anywhere that issues a chargeback, posts a credit, or records a final liability decision.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-20. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-20 — r001-rcv-disc. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with no key configured renders the whole panel -- every case's lines and its carrier exception note, since data/cases.jsonl and data/gold.jsonl are committed, not fetched (python -m src.app). Clicking Resolve with no API_KEY returns a calm 200 explaining nothing was called (src/app.py's /api/check). It cannot reproduce a determination, a score, or a dollar figure without a key -- those are what results/eval-r001-rcv-disc.json already committed.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per case: one HTTP request carrying every SKU line (PO/BOL/receiving quantities) and the carrier-exception note (or its absence), parsed to a liable party, discrepant SKU, cited quantity pair, delta and drafted notice. No retrieval step -- loading a case from disk is pure code and costs no network time. This run left provider-side reasoning ('thinking') at its documented default (on) rather than disabling it, so p50/p95 include a hidden reasoning pass on every call, not only the JSON-formatting cost -- see not_good_enough.

Current processWhat it replaces

Receiving/AP staff reading a receiving-discrepancy case's PO, BOL and warehouse-scan quantities line by line, checking whether any carrier exception note on file actually names the discrepant SKU before proposing who is liable -- for every case, every day.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• jsonl
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one characters per case's assembled block is

One case, its 2-4 SKU lines and its one carrier-exception field (or none) included whole -- no train/test split, every case resolved once, in one call.

SetupWhat the setup figure measured

No index is built. src/app.py finds a case by scanning R.cases() for a matching case_id -- 36 rows, a linear scan, not a search structure -- and src/resolve.py sends that case's full record whole into the one call. build_seconds and build_cost_usd are both zero because there is no index-build step, not because one ran for free.

LicenceLicence

MIT (this repository's own licence)

Bring your ownBring your own cases

Point tools/build_corpus.py's CASE_PLAN, CARRIERS and EXCEPTION_TEMPLATES at your own SKUs, carriers and case shapes, or write your own data/cases.jsonl in the same shape -- src/resolve.py and src/prompt.py read a case by its own fields (lines[].sku/po_qty/bol_qty/received_qty, carrier_exception) and do not care where the numbers came from. The four-value taxonomy in src/prompt.py assumes exactly these four liable parties; a different set is a prompt.py AND tools/build_corpus.py's classify_case() change, not a data-only change.

⚠︎ And what stops being true when you do: The measured 100% liable_party accuracy and 0% false_confident_call/wrong_sku_exception_misread rates are THIS corpus's exception-note phrasing (6 hand-written templates) and THIS corpus's trap design (6 of 14 internal cases, ~43%, planted on purpose). A real carrier's own exception-note phrasing, and a real receiving operation's own missing-document shape (see breaks_on), are both unmeasured by this run.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — but they are the input total apportioned by each part’s share of the characters, not a per-part reading from the provider’s counter. The total is measured; the split is arithmetic, and a part carrying denser text really costs more of it than this says.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 972 tokens assembled, 915 are instructions — 94% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Logged verbatim by importing src.prompt and src.resolve directly and calling prompt.build(case) for CASE-00001 (not retyped) -- the literal system and user content src/resolve.py::check() sends. Per-part token counts are a proportional estimate over character share (system 3117 / case 195 chars, 3312 total) applied to this call's own total of 972 input tokens -- the provider reports only the call's total, matching lenses.LLM.tokens.input exactly, never a per-segment split. The 'case' part covers only the case's lines and exception-note block src/prompt.py::_case_block() builds -- the surrounding 'Build the case file...' instruction text is part of the user message actually sent (see prompt_verbatim) but is not a separately named segment in the kit's own code.

Full promptThe prompt in full

You build a receiving-discrepancy case file for one case and propose a probable liable party, for receiving/AP to confirm before any chargeback or credit is issued. You are given every SKU line on the case -- each with the quantity the PO ordered, the quantity the BOL states was shipped, and the quantity receiving actually scanned -- plus one carrier exception note, which is either absent or names ONE specific SKU.

Exactly one line on the case is genuinely discrepant: its PO, BOL and receiving quantities do not all agree. Every other line is clean (all three quantities equal) -- find the discrepant line among them, do not assume it is the first one listed.

THE FOUR VALUES FOR liable_party, EXACTLY ONE APPLIES
  vendor                 the BOL states LESS than the PO ordered, and receiving recorded exactly what the BOL says was shipped. The carrier delivered everything it was given; the vendor gave it less than the PO called for. Cite PO vs BOL.
  carrier                the BOL states the full PO quantity was shipped, receiving recorded less than that, AND the carrier's own exception note names this exact SKU. The carrier picked up the full order and delivered less, and its own paperwork corroborates a transit loss or damage for this line. Cite BOL vs Received.
  internal               the BOL states the full PO quantity was shipped and receiving recorded less than that, but nothing corroborates a transit loss for THIS line -- either there is no carrier exception on file at all, or the exception on file names a DIFFERENT SKU than this one. It looks like a transit-loss pattern, but nothing on file backs it for this line; the likely explanation is a dock miscount, not a proven transit loss. Cite BOL vs Received.
  insufficient_evidence  the three quantities disagree in a shape none of the other three calls resolve cleanly -- most often the BOL and receiving AGREE with each other but both disagree with the PO (an apparent over-shipment, not a shortfall), or more than one of the three quantities is off at once. There is no clean documentary basis for a liability call. Cite PO vs Received.

⚠︎ A CARRIER EXCEPTION EXISTING ON THE CASE IS NOT, BY ITSELF, EVIDENCE THAT THE CARRIER IS LIABLE FOR THIS LINE. Check whether the exception names the SAME SKU as the discrepant line. An exception about a different SKU is a real note about a different problem -- it does not excuse this one, and calling the carrier liable on its strength alone is exactly the mistake this task exists to catch.

The notice you draft must name the discrepant SKU and state which two of the three quantities (po, bol, received) disagree, their values, and the delta between them -- e.g. "BOL states 240 units shipped; receiving recorded 210 units, a shortfall of 30." Use whichever pair the liable_party value above says to cite. delta is the absolute difference between qty_a and qty_b, never signed.

You never issue a chargeback or a credit yourself, and you never make the final liability determination -- you propose a probable liable party and its documentary basis for receiving/AP to confirm and act on.

Build the case file for the case below and propose a probable liable party.

Return a JSON object with exactly these keys: "liable_party" (one of: vendor, carrier, internal, insufficient_evidence), "discrepant_sku" (the SKU of the one discrepant line), "doc_a" and "doc_b" (the two of "po", "bol", "received" that disagree, per the liable_party value's citation rule above), "qty_a" and "qty_b" (their quantities), "delta" (the absolute difference between qty_a and qty_b), "notice_text" (the drafted notice, one or two sentences).

CASE CASE-00001 (PO PO-356479, carrier Union Point Freight)
Lines:
  SKU-4001    PO 334   BOL 334   Received 334  
  SKU-4002    PO 105   BOL 124   Received 124  
Carrier exception: none on file

Raw responseThe raw response

{
  "liable_party": "insufficient_evidence",
  "discrepant_sku": "SKU-4002",
  "doc_a": "po",
  "doc_b": "received",
  "qty_a": 105,
  "qty_b": 124,
  "delta": 19,
  "notice_text": "PO ordered 105 units of SKU-4002; receiving recorded 124 units, an overage of 19. BOL also states 124 units, so this appears to be an over-shipment rather than a transit loss."
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check who's liable when a retailer's delivery doesn't add up — 36 cases. One model answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

The grader is evals/scoring.py and it is pure code: exact match on liable_party (the core 4-way classification); false_confident_call (a confident, non-insufficient_evidence call on a gold insufficient_evidence case) counted over exactly that denominator; wrong_sku_exception_misread (a trap case credited to the carrier anyway) counted over exactly the corpus's planted trap denominator; and discrepant_sku plus quantity-citation accuracy checked separately, requiring the same document pair and delta, not just two correct numbers. No judge model. The same function scores both evals/baseline.py's free floor and evals/run.py's real run.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran r001 publishes no rate card this repo commits, so nothing here is what was actually paid -- the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page. Reasoning was also left at provider default (on) for this run -- see Business.not_good_enough -- so even the projected figure prices tokens the shipped app's own reasoning-off configuration would not spend.

Grading unitWhat the grading figure prices

Grading calls nothing -- exact match on liable_party/SKU/doc-pair/delta is pure code. A measured $0.00, not an unpriced one.

The gradersOne way to grade, and why it is the only one

The floor blames the carrier on ANY exception note present on the case, without checking which SKU it names -- so it misclassifies all 6 planted trap cases as carrier when gold says internal (a 100% wrong_sku_exception_misread rate, 6 of 6), while getting every other case right: the corpus never plants an exception note on a vendor or insufficient_evidence case, and the 8 internal_clean cases carry no exception at all, so presence-of-exception and correct-SKU-match agree everywhere except the 6 planted traps. Its overall liable_party accuracy still reaches 83.3% (30 of 36) for exactly that reason. The fast tier's 0% wrong_sku_exception_misread and 100% liable_party accuracy on the identical 36 cases is the real, measured gap a model closes that this floor cannot.

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The exception-first baseline (evals/baseline.py) and the scorer (evals/scoring.py) are both pure code and cost $0.00 to run against either result set. The figure above is this run's own 35907 input / 19706 output tokens priced at Google Gemini 3 Flash's published rate, the same basis cost_per_query_usd uses, not a second, larger spend.

Cost driversWhat actually moves the bill

• Every SKU line on the case (PO, BOL and receiving quantities) plus the carrier exception note or its absence -- fixed size per call by construction (input tokens ranged 971-1024 across all 36 calls, essentially flat), the floor every call pays regardless of which liable party governs.
• Reasoning ('thinking') was left at the provider's default (on) rather than explicitly disabled for this run -- it consumed 76.9% of the average output-token budget (421.1 of 547.4 tokens/call) and is already priced into cost_per_query_usd, since a provider bills reasoning tokens as completion tokens. src/app.py's live UI hardcodes thinking off, a different, unmeasured configuration -- see Business.not_good_enough.
• The fixed system prompt (3,117 characters -- the four-value taxonomy, citation vocabulary and the wrong-SKU-exception warning) is sent in full on every call, the largest single fixed cost per case.

Your volumeWhat it costs at your volume

Linear in cases: each call is independent and self-contained, with no shared context or retrieval step to amortise. This run's 36 cases cost about $0.0771 projected onto Google Gemini 3 Flash's published rate, so ten times the set is about $0.7710 on the same rate and the same reasoning-on configuration -- arithmetic on the measured per-call rate, not a second run.

Model choiceWhy this model

The fast tier was the only model run against this corpus. No second tier was run -- unlike data-reconcile's fast/reasoning comparison -- so this page prices one model, not a trade-off; see Eval.could_not_verify for what a second run would need to answer.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 36 receiving-discrepancy cases resolved, 36 cases scored by pure code. This run (r001-rcv-disc) answered 36 of 36 liable-party calls -- the row this table prices, the same one Cost.cost_by_model[0] uses.

The codeEvery file, and what it is for

Eight modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pycorpus generator — a swap seam

Generates 36 receiving-discrepancy cases from a fixed seed (SEED=20260819) -- 2-4 SKU lines per case, one genuinely discrepant, plus an optional carrier exception note naming exactly one SKU. Gold (liable_party, discrepant_sku, doc_a/qty_a, doc_b/qty_b, delta, trap) is RECOMPUTED from the actual generated case record by classify_case(), never from the target category the generator aimed for -- --verify independently re-checks every row against the record actually written to disk.

You change it to: Point it at your own SKUs, carriers and case shapes. src/resolve.py and src/prompt.py read a case by its own fields (lines[].sku/po_qty/bol_qty/received_qty, carrier_exception) and do not care where the numbers came from.

# Generate the receiving-discrepancy cases this kit checks against, from a fixed seed.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
SEED = 20260819
DOCS = ("po", "bol", "received")
CASE_PLAN = (
CARRIERS = [
EXCEPTION_TEMPLATES = [
def classify_case(lines, carrier_exception):
def _clean_line(sku, rng):

src/prompt.pythe prompt

The four-value liability taxonomy and the citation vocabulary, declared once and read from here by build() and parse(). States the wrong-SKU-exception failure mode to the model explicitly, as the named failure this task exists to catch, not left for the model to discover.

# Assemble the one prompt this kit sends, and parse the one reply it gets back.

LIABLE_PARTIES = ("vendor", "carrier", "internal", "insufficient_evidence")
DOCS = ("po", "bol", "received")
LIABLE_MEANINGS = {
SYSTEM = (
DEFAULT_PROMPT = "v1"
SYSTEMS = {"v1": SYSTEM}
def _case_block(case):
def build(case, prompt=DEFAULT_PROMPT):
def parse(raw):

src/resolve.pythe AI layer

Loads one case by id, calls the model once with every SKU line and the carrier exception note (or its absence), parses the reply into a liable party, discrepant SKU, cited quantity pair, delta and drafted notice. The whole AI layer, deliberately short -- the same split data-reconcile's reconcile.py and fin-payrun's payrun.py both make. Never issues a chargeback or credit and never makes the final liability call -- there is no function here or in src/app.py that does either.

# Build one receiving-discrepancy case file and propose a probable liable party: load the case,

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CASES = os.path.join(HERE, "data", "cases.jsonl")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
MAX_TOKENS = 3000
def cases():
def load_gold():
def check(cfg, case, complete=None, thinking=None, prompt=P.DEFAULT_PROMPT):

src/adapters/__init__.pythe model adapter — a swap seam

SEAM -- the model. Raw HTTP for every provider (openai-compatible, anthropic), stdlib only. Passes a thinking kwarg through to the provider only when the caller supplies one, and reports token_details.reasoning_tokens when the provider returns it -- the field the reasoning-token finding in Business.not_good_enough is measured from.

You change it to: .env plus the same run again. Any OpenAI-compatible provider or Anthropic.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/budget.pyspend control

An append-only ledger written BEFORE each call, shared by every kit under one .env so the daily call cap is on the KEY rather than per kit. Counts calls, not dollars, on purpose -- see the file's own docstring.

# A cap on live calls, shared by every kit on this machine. No dependency, no service.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
KIT = os.path.basename(HERE)
ROOT = os.path.dirname(os.path.dirname(HERE))
SHARED_ENV = os.path.join(ROOT, ".env")
LEDGER = os.path.join(ROOT if os.path.exists(SHARED_ENV) else HERE, ".calls-ledger.jsonl")
PAID_ARMS = ("r", "x")
FREE_ARMS = ("t", "b")
BOOT = os.urandom(6).hex()
PID = os.getpid()

src/app.pythe app

Static files and three JSON endpoints (/api/state, /api/case, /api/check) on the standard library. /api/check hardcodes thinking=THINKING_OFF on every live call -- a different reasoning setting from the one the registered eval run (r001-rcv-disc) actually used; see Business.not_good_enough.

# The minimal local UI. Standard library only -- python -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "8790"))
class H(BaseHTTPRequestHandler):
def main():

evals/scoring.pythe scorer

Four-way liable_party exact match, pure code: false_confident_call (a confident call on a gold insufficient_evidence case) and wrong_sku_exception_misread (a trap case credited to the carrier anyway) reported as their own named figures rather than folded into a generic 'wrong' bucket, plus discrepant_sku and quantity-citation accuracy checked separately.

# Score a set of predicted liable-party determinations against gold. Pure code, shared by

LIABLE_PARTIES = ("vendor", "carrier", "internal", "insufficient_evidence")
def _pct(n, d):
def score(records, gold):

evals/baseline.pyfree baseline

A fixed rule, written the way a person free-texting a quick classifier would write one, that blames the carrier on ANY exception note without checking which SKU it names -- so it misclassifies every planted trap case by construction. The honest, narrow floor a model has to clear.

# What a simple, dumb rule catches, over the same corpus. Free. No key, no dependency, no model

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
def classify(case):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 997 input and 547 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's cases are entirely synthetic (tools/build_corpus.py) -- no untrusted party wrote any text the model actually read this run. In a real deployment the carrier exception note would arrive from an external carrier's own EDI feed or claims portal message -- exactly the kind of externally-authored input data-reconcile's agreement-poisoning attack measures for a different artifact, and this kit's architecture treats it as trusted input with no verification step. No attack has been tried against this kit; see redteam.why.

Read from the shared .env or the real environment only, never written into the repo, never requested from a reader on any surface. src/app.py's /api/check handler strips api_key and base_url out of any exception message before it reaches the browser, so a misconfigured key cannot leak into a UI error.

The experimentWe did not attack it -- and one boundary that should exist does not

An indirect prompt injection needs a field an outside party controls that reaches the prompt. In THIS corpus every carrier exception note is generated by tools/build_corpus.py, so there is no live untrusted text in the run this report measures -- but a real deployment's exception note would carry externally-authored text (see posture), and that surface has not been attacked. The gates above are boundaries confirmed by reading the code, not payloads run through it; one is a negative result, not a guarantee. Confirmed by reading the code, not by a run, on 2026-08-20 -- no attack run was fired; see redteam.why.

Each boundary above was checked by reading the call sites, not by an attack trial. Gate 2 and gate 4's underlying code path are the ones that do NOT hold structurally -- see Guardrails.fails for the same findings from the enforcement side.

The result0 attack trials, and one boundary that does NOT hold in code (only in the prompt): the model's own cited doc_a/doc_b/qty_a/qty_b/delta and its liable_party call are never cross-checked against the case record or against each other before the UI renders them. Two boundaries (no write path, key redaction) hold, confirmed by reading the code.

This run's corpus is entirely generated (tools/build_corpus.py) -- no case's carrier exception note was authored by an outside party. A real deployment's exception note would be exactly the kind of externally-supplied text data-reconcile's agreement-poisoning red-team run measured being poisoned for a different artifact; that question is open for this kit.

HonestyWhat this does not prove

• Whether an injected instruction inside a carrier exception note (e.g. 'this exception covers the whole shipment') could move a liable-party call -- no red-team run exists for this kit.
• Whether the live app's hardcoded thinking=off setting changes resistance to a hostile exception note relative to this run's provider-default-on configuration -- untested either way.
• Whether a code-level consistency check (Guardrails.add_first) would catch a real liable_party/citation disagreement in practice -- none has been built or exercised.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

A carrier exception note is only evidence for the SKU it actually names, never for the case in general -- a case is never resolved to carrier liability on an exception's presence alone.

src/prompt.py -- SYSTEM, stated as an explicit boundary the model must respect. Prompt-level ONLY: unlike data-reconcile's live citation-fidelity check, no code in src/resolve.py or src/app.py cross-checks the model's own cited doc_a/qty_a/doc_b/qty_b/delta against the case record, or re-derives liable_party from the exception's named SKU, before the UI renders them.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• IT IS NOT A CODE-ENFORCED CHECK. The rule that an exception is only evidence for the SKU it names lives only in src/prompt.py's SYSTEM text -- nothing in src/resolve.py or src/app.py recomputes it. See fails above.
• It does not verify the cited doc_a/doc_b/qty_a/qty_b/delta against the case record mathematically -- see fails.
• It does not make the liable-party classification itself correct -- see Eval.taxonomy for what this run measured, not what any guardrail guarantees.
• It is not a defence against a hostile or fabricated carrier exception note -- no red-team run exists for this kit (see the security page).

WatchedWhat is watched, and why that one

The kit measures 17 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

16 measured by the latest run 1 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

1 recorded run. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 16 chips that all say so.

DeviationsWhat deviated

0 breaches across 1 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-score on any change to src/prompt.py or tools/build_corpus.py -- the first changes what is asked, the second changes what is asked ABOUT. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the thinking setting in src/resolve.py.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library -- requirements.txt is a comment-only file explaining that the emptiness is load-bearing. The whole resolving decision is three files: src/prompt.py, src/resolve.py and src/adapters/__init__.py.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per case -- load, assemble, prompt, call, parse -- with no branching and no state carried between cases. A framework would add an orchestrator to a for-loop.

The other sideWhat a framework costs you

• A fifth liable-party value this kit's fixed four-value vocabulary doesn't name needs a hand-written LIABLE_PARTIES/LIABLE_MEANINGS entry plus new corpus category logic in tools/build_corpus.py, rather than being configured declaratively.
• No built-in retry/backoff beyond src/adapters.py's own bounded retry -- a framework's queue and worker model is not here, so a real deployment adds its own scheduling.
• No built-in observability beyond what evals/run.py prints and writes to results/ -- a framework's tracing/dashboard integration is not here.
• No built-in output validation beyond src/prompt.py::parse()'s own tolerant-but-not-creative JSON parse -- a framework with a schema-and-consistency-check layer baked in might catch a liable_party/citation disagreement before it renders; this kit's stdlib-only design did not build one (see Guardrails.fails).

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-rcv-disc on the fast tier, 2026-08-20. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. This is the only run on record, so there is nothing to move against. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT (this repository's own licence) Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineLiable-party exact match, plus false-confident-call, wrong-SKU-trap and quantity-citation checks

Does the model's liable_party match gold's four-way classification? Does it avoid a confident call on a case gold says has no clean documentary basis (false_confident_call)? Does it avoid crediting a carrier exception note that names a different SKU than the discrepant line (wrong_sku_exception_misread)? Do the cited discrepant SKU, document pair, quantities and delta match gold?

MethodHow the test was run

evals/scoring.py::score, in-process, no key and no model. The same function evals/baseline.py and evals/run.py both call -- a baseline and a real run scored by two different scorers cannot be compared honestly.

The inputOne real row, seen by every grader

The formulaWhat it computes

liable_accuracy = correct liable_party / answered cases, over answered=36. false_confident_call = a non-null, non-insufficient_evidence call on a gold insufficient_evidence case, counted over exactly that denominator (6). wrong_sku_exception_misread = a call of 'carrier' on a gold trap case (exception on file naming a different SKU), counted over exactly the planted-trap denominator (6). discrepant_sku_accuracy and quantity_citation_accuracy are their own exact-match rates, the latter requiring the same (doc, qty) pair as an unordered set plus a matching delta.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's classify_case(), which reads the same case record IN ORDER and computes the gold liable_party, discrepant_sku, doc_a/qty_a/doc_b/qty_b, delta and trap flag -- re-derived independently via --verify against the case records actually written to disk, asserted zero drift.

How tight can the band be? There is no tolerance band here -- every field is an exact match (liable_party, SKU string, doc labels, numeric delta); the liability-resolution task has no continuous quantity to round.

Cadence: Re-run on any change to src/prompt.py, tools/build_corpus.py, or MAX_TOKENS in src/resolve.py -- the first changes what is asked, the second changes what is asked ABOUT, the third bounds how many verdicts can come back at all.

The decisionWhen to reach for it