The business caseThe problem this solves

Before a bulk tank is loaded, somebody has to decide whether the residue of its prior cargo is compatible with the product about to go in — or whether the tank needs a specific cleaning regime first, or must be refused outright. It is a compatibility lookup plus reasoning over a cleaning record, and the sheet puts two things next to the answer that look like evidence and are not: the tank log's own claim about the wash it was given, and the terminal inspector's free-text remark at the hatch. Only the CERTIFICATE counts, and the note is nobody's measurement. Someone opening each pre-load check sheet, reading the incoming product and the grade it is being loaded as, looking the prior cargo up in a compatibility chart, checking whether the grade carries a restricted-predecessor rule and how many cargoes back it reads, then working out whether the cleaning CERTIFICATE on file actually covers the regime that pairing needs. It is a few minutes per tank and it is every tank, and the part that goes wrong is not the chart lookup: it is reading the tank log's claim instead of the certificate, stopping at the prior cargo when the grade reads two back, or treating a predecessor ban as something a thorough wash can cure.

The inputThe actual pre-load check sheets

The corpus is 55 pre-load check sheets, 0.03 MB (txt 55). Plain text, one format, invented rather than fetched — a real pre-load check sheet names a real tank on a real berth, a real customer's product, a named inspector's own remark and a real carrier's cleaning history, and there is no public corpus of (pre-load sheet, correct verdict) pairs for the same reason there is no public corpus of bank statements. Generating it also makes the label mechanical: gold's verdict is the matrix lookup over the same five values the sheet states, never somebody's reading of the note.

Tank
----
TNK-AE-71805

Terminal and Berth
------------------
Lowdock Terminal, Berth 1

Incoming Product
----------------
ethanol

Incoming Grade
--------------
food_grade

Prior Cargo
-----------
methanol

Two-Back Cargo
--------------
none -- tank recertified before the prior cargo

Wash Performed
--------------
hot_detergent_wash

Cleaning Certificate
--------------------
CLC-35824 -- certified for: hot_detergent_wash

Load Status
-----------
pending

Inspector Notes
---------------
Something looked off in the tank record, flagged for a second look before loading.

The outcomeWhat a good result looks like

A ten-field extracted record per sheet, plus one pure-code hold decision taken from two of those fields: a tank that is not clear to load as it stands, whose product has already gone in, is the one to quarantine today. The verdict carries the matrix's own reasoning and names what it could not determine. Nothing here authorises a load, releases a tank, or contacts anybody.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Screening a day's pre-load check sheets before a vetting or terminal operations desk opens them — the fast tier — it ties the deliberating tier at 55 of 55 verdicts with 0 unsafe releases, measured three times across the two
  100 pct four-way verdict accuracy on both runs against the free inspector-note floor's 36.36 pct, and 0 unsafe releases against the floor's 15 of 39.
• Deciding the two tiers on cost, speed or accuracy — the fast tier
  12.1 pct cheaper per sheet ($0.0025141 against $0.0028613), 50.0 pct lower p50 latency (4516 ms against 9024 ms), and identical on every one of the three published graders — 550 of 550 cells, 55 of 55 verdicts, 0 unsafe releases, 19 of 19 hold flags, on both.

And where nothing here is good enough:

• Deciding whether this shape is worth buying at all, from these numbers — nothing yet — this corpus cannot answer it
  Three runs across two tiers and a procedure-stripped ablation all scored perfectly, so nothing here separates any two configurations. What the numbers DO support is narrower and real: a tone read of the inspector's note is measurably unsafe on this shape of decision, and a self-consistency check catches all 35 of that shortcut's errors with no labels.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-cargo-compat. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts, the 55-sheet picker populates, the field table draws its ten empty rows and the shipped matrix renders in full from data/matrix.json; clicking Check returns the no-key sentence rather than a stack trace, and the hold row reads “not computed — one of the two values the rule needs was missing” rather than “no”. python -m evals.check_labels passes with no network access at all.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per pre-load check sheet

Current processWhat it replaces

Someone opening each pre-load check sheet, reading the incoming product and the grade it is being loaded as, looking the prior cargo up in a compatibility chart, checking whether the grade carries a restricted-predecessor rule and how many cargoes back it reads, then working out whether the cleaning CERTIFICATE on file actually covers the regime that pairing needs. It is a few minutes per tank and it is every tank, and the part that goes wrong is not the chart lookup: it is reading the tank log's claim instead of the certificate, stopping at the prior cargo when the grade reads two back, or treating a predecessor ban as something a thorough wash can cure.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 0 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings; a sheet with none falls back to one whole-document segment

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 55 sheets cut into 550 sections in under a hundredth of a second, in process, with no model and no network.

LicenceLicence

MIT — this repository's own licence. Every tank id, terminal name and inspector note is invented and every cargo is an ordinary generic chemical or oil name; no trade name, supplier, carrier or terminal operator is named. The compatibility matrix this kit ships (data/matrix.json) is ILLUSTRATIVE and is not an authority. It was written for this kit and reproduces no commercial, industry-body or proprietary chart, no carrier's or terminal operator's own prior-cargo list and no regulatory schedule.

Bring your ownBring your own pre-load check sheets

REPLACE data/matrix.json FIRST. It is this kit's own construction, it resembles no real compatibility chart, and everything downstream reads it: src/matrix.py loads it, src/prompt.py renders it into every call, and tools/build_corpus.py writes gold with it. Then replace data/corpus/*.txt, write your own data/fields.json, and supply a gold record per sheet. SECTION_HINTS in src/select.py maps fields to section headings and will need editing for a different sheet layout; when it does not match, selection falls back to the whole document — slower, more expensive, always correct. compute() in src/extract.py is this kit's own hold rule and is the second thing to replace.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 2,093 tokens assembled, 796 are instructions — 38% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on CGO-0017 — the same sheet the worked example above was captured on, so the four parts sum to exactly the 2,093 input tokens that call actually reported. Four calls at max_tokens=1, each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. kits/UC0048-cargo-compat/results/tokens-p002-cargo-compat.json. ⚑ THE SAME MEASUREMENT ON A DIFFERENT SHEET (CGO-0001, tokens-p001-cargo-compat.json) returned 796 / 473 / 640 for the first three parts — byte-identical, which is the evidence that the 1,909-token fixed prefix really is fixed and is paid on every call.

Full promptThe prompt in full

You read a bulk-tank PRE-LOAD COMPATIBILITY CHECK SHEET and extract structured fields from it. You return JSON and nothing else.

You are PROPOSING a verdict for a qualified person to authorise. You never authorise a load, and your verdict is not a substitute for the incoming product's safety data sheet or for a competent person's assessment.

RULES, in order of importance:
1. If the sheet does not state a field, return null for it. Do not infer it and do not use what you know about the world.
2. `verdict` is decided ONLY by the COMPATIBILITY MATRIX given below, applied to five values -- incoming_product, incoming_grade, prior_cargo, two_back_cargo and wash_certified_for. Work through these four checks IN ORDER and STOP at the first one that fires:
   a. NOT KNOWN. If the sheet does not record a prior cargo, or if the incoming product or either recorded cargo is not listed in the matrix, answer 'undetermined'. Never guess a class for a cargo the matrix does not carry.
   b. REACTIVE PAIR. If the prior cargo's class and the incoming product's class are listed as a reactive pair, answer 'refuse'. The pair is symmetric, and no cleaning regime clears it.
   c. RESTRICTED PREDECESSOR. Read the look-back depth for the incoming grade. Technical grade reads the prior cargo only. Food grade and high purity read the prior cargo AND the two-back cargo. If ANY cargo in that chain is banned for this grade -- by its class or by name -- answer 'refuse'. A BAN IS NOT A CLEANING PROBLEM: cleaning does not cure it, however thorough the certificate is.
   d. MINIMUM CERTIFIED WASH. Take the minimum wash for the PRIOR cargo's class, then raise it one rung if the incoming grade is food_grade or high_purity (capped at the top of the ladder). Compare it against what the CERTIFICATE names. Meets or exceeds it: answer 'accept'. Falls short: answer 'clean_then_load'.
3. THE CERTIFICATE GOVERNS, NOT THE TANK LOG. `wash_performed` is what the log claims was done and it is NEVER an input to the verdict. `wash_certified_for` is what a certificate actually covers, and it is the only wash the tank is credited with. When the two disagree, use the certificate. A tank with no certificate on file is credited with NO wash at all, whatever its log says.
4. THE INSPECTOR'S NOTE IS A FIELD TO COPY, NOT EVIDENCE ABOUT COMPATIBILITY. A note that sounds worried does NOT make a compatible pair incompatible, and a note that sounds relaxed does NOT clear a tank. The matrix and the certificate decide; the note is one person's remark and may disagree with them.
5. Copy cargo and product names verbatim from the sheet, in the lower case it states them. Where the sheet says the prior cargo is 'not recorded', return null for prior_cargo rather than the words. Where it says the tank was recertified and has no cargo before the prior one, return null for two_back_cargo.
6. Use the exact allowed value for a field that lists them.
7. Return every field named in the schema, even when the answer is null.

COMPATIBILITY MATRIX (the authority for `verdict`; this is an ILLUSTRATIVE matrix shipped with this kit, not an industry chart)

CARGO CLASSES
  acid:        sulphuric acid, hydrochloric acid, acetic acid, phosphoric acid
  alcohol:     methanol, ethanol, isopropanol, ethylene glycol, propylene glycol
  caustic:     sodium hydroxide solution, potassium hydroxide solution
  food_oil:    refined soybean oil, refined sunflower oil, refined rapeseed oil, refined coconut oil
  mineral_oil: base oil, gas oil, white mineral oil
  oxidiser:    sodium hypochlorite solution, hydrogen peroxide solution
  solvent:     toluene, xylene, acetone, methyl ethyl ketone

REACTIVE PAIRS -- refuse outright, symmetric, no cleaning clears them
  acid + caustic
  acid + oxidiser
  oxidiser + solvent
  oxidiser + alcohol

LOOK-BACK DEPTH, by incoming grade
  technical:   1 cargo back (the prior cargo only)
  food_grade:  2 cargoes back (the prior cargo AND the cargo before it)
  high_purity: 2 cargoes back (the prior cargo AND the cargo before it)

BANNED PREDECESSORS -- refuse; cleaning does not cure a ban
  technical:   none
  food_grade:  any cargo of class solvent, mineral_oil, oxidiser; plus by name: methanol
  high_purity: any cargo of class solvent

WASH LADDER, weakest to strongest
  none < water_rinse < hot_detergent_wash < caustic_wash < steam_and_dry

MINIMUM CERTIFIED WASH, by the PRIOR cargo's class (then raise one rung for a food_grade or high_purity load, capped at the top of the ladder)
  acid:        water_rinse
  alcohol:     water_rinse
  caustic:     water_rinse
  food_oil:    hot_detergent_wash
  mineral_oil: caustic_wash
  oxidiser:    water_rinse
  solvent:     caustic_wash

Extract these fields:
- tank_id (string) -- the tank or container identifier, verbatim
- incoming_product (string) -- the product about to be loaded, verbatim and in lower case as the sheet states it
- incoming_grade (enum) one of: technical, food_grade, high_purity -- the grade the incoming product is being loaded as, verbatim
- prior_cargo (string) -- the last cargo this tank carried, verbatim and in lower case. When the sheet says the prior cargo is 'not recorded', return null -- do not guess it and do not copy the words 'not recorded'
- two_back_cargo (string) -- the cargo carried BEFORE the prior one, verbatim and in lower case. When the sheet says there is none because the tank was recertified, return null
- wash_performed (enum) one of: none, water_rinse, hot_detergent_wash, caustic_wash, steam_and_dry -- the wash the tank log claims was carried out after the prior cargo, verbatim. Report it as stated -- it is NOT what decides the verdict
- wash_certified_for (enum) one of: not_certified, water_rinse, hot_detergent_wash, caustic_wash, steam_and_dry -- the wash regime the cleaning CERTIFICATE actually certifies, taken from the Cleaning Certificate section. When the sheet says there is no certificate on file, return 'not_certified'. This may name a different regime from wash_performed, and when it does, this is the one that counts
- load_status (enum) one of: pending, loaded -- has the incoming product already gone into this tank, or is the load still pending?
- inspector_notes (string) -- the terminal inspector's own free-text remark on the tank, copied verbatim
- verdict (enum) one of: accept, clean_then_load, refuse, undetermined -- the pre-load compatibility verdict this sheet's own facts produce. Decide it STRICTLY from incoming_product, incoming_grade, prior_cargo, two_back_cargo and wash_certified_for -- never from inspector_notes and never from wash_performed. Work through the four checks IN ORDER, stopping at the first that fires. (1) If prior_cargo is not recorded, or either cargo is not in the shipped matrix, answer 'undetermined'. (2) If the prior cargo's class and the incoming product's class are a reactive pair, answer 'refuse'. (3) Read the predecessor ban over the look-back depth for the grade -- technical reads the prior cargo only, food_grade and high_purity read the prior cargo AND the two-back cargo -- and if any cargo in that chain is barred for this grade, answer 'refuse'; cleaning does not cure a ban. (4) Otherwise compare the wash the CERTIFICATE names against the minimum for the prior cargo's class at this grade: meets or exceeds it, answer 'accept'; falls short, answer 'clean_then_load'

Return a JSON object with exactly these keys: tank_id, incoming_product, incoming_grade, prior_cargo, two_back_cargo, wash_performed, wash_certified_for, load_status, inspector_notes, verdict
Use null for any field the sheet does not state.

PRE-LOAD CHECK SHEET
--------------------
Tank
----
TNK-WP-42393

Incoming Product
----------------
refined sunflower oil

Incoming Grade
--------------
food_grade

Prior Cargo
-----------
refined rapeseed oil

Two-Back Cargo
--------------
gas oil

Wash Performed
--------------
caustic_wash

Cleaning Certificate
--------------------
CLC-53016 -- certified for: caustic_wash

Load Status
-----------
loaded

Inspector Notes
---------------
Standard changeover, nothing unusual about this unit at all.

Raw responseThe raw response

{
  "tank_id": "TNK-WP-42393",
  "incoming_product": "refined sunflower oil",
  "incoming_grade": "food_grade",
  "prior_cargo": "refined rapeseed oil",
  "two_back_cargo": "gas oil",
  "wash_performed": "caustic_wash",
  "wash_certified_for": "caustic_wash",
  "load_status": "loaded",
  "inspector_notes": "Standard changeover, nothing unusual about this unit at all.",
  "verdict": "refuse"
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check a bulk tank's past cargoes before loading new product — 55 pre-load check sheets. Two tiers of one model family answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran these runs publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersThree ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The inspector-note floor (evals/baseline.py) and the scorer (evals/judge.py) are both pure code and cost $0.00 to run against any result set. The figure above is the THREE SCORED runs' own token counts (55 sheets each — two fast tier, one deliberating) priced at Google Gemini 3 Flash's published rate — the same basis cost_per_query_usd uses. ⚠︎ IT IS NOT WHAT BUILDING THIS KIT COST. Four further paid passes sit outside it and are committed in results/: a 6-sheet calibration that MEASURED max_tokens ($0.0151), the superseded 55-sheet pass lost to the adapter's transport defect ($0.1333), the 26-sheet procedure-ablation ($0.0688), and 8 prompt-split calls plus one worked example ($0.0117). All 262 live calls together project to $0.6638 on this card.

Cost driversWhat actually moves the bill

• The fixed prefix — system prompt (796 tokens), compatibility matrix (473) and field schema (640) — is 1,909 of 2,100 tokens on the example call, 91 pct, and is byte-identical on every one of the 55 calls. The sheet itself is 191 tokens. The bill is the matrix and the instructions, not the document.
• Provider-side reasoning. 20,553 of r001's 26,889 output tokens (76.4 pct) were reasoning rather than the JSON record, on a ten-key reply that is about 190 tokens of actual content. Reasoning was left at the provider's default and nothing here has measured what turning it off would cost in accuracy.
• Output length: the model returns a full ten-key JSON record every call, including the inspector's note copied back verbatim, whatever the sheet says.

Your volumeWhat it costs at your volume

Linear in check sheets: each call is independent, carries the same fixed prefix and shares nothing with its neighbours, so 550 sheets cost ten times 55 and take ten times as long. Nothing amortises — there is no index and no cache. ⚠︎ AND THE MATRIX DOES NOT SCALE LINEARLY, IT SCALES WORSE: it is sent whole in every call, so a chart of 500 cargoes rather than 25 multiplies the input side of every call at once. At that size the lookup has to move into code before the call, with only the relevant rows sent.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 55 bulk-tank pre-load check sheets, one completion call each, on the fast tier. The deliberating tier's own 55 calls and the fast tier's repeat are recorded separately in Cost.cost_by_model and are not projected here.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the pre-load check sheet into addressable sections, pure code

# Cut a document into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections carry each field, pure code — the verdict is mapped to the five facts the matrix actually reads, never to the tank log's wash claim and never to the inspector's note, and the Terminal and Berth section is mapped by nothing and never sent

You change it to: map fields to your own check sheet's headings; unmatched falls back to the whole document

# Pick which sections plausibly carry each field. Pure code -- the last deterministic step

SECTION_HINTS = {
def for_field(secs, field):
def plan(secs, fields):

src/matrix.pymatrix

the compatibility lookup, loaded from data/matrix.json — classes, reactive pairs, the grade-driven predecessor ban with its look-back depth, and the minimum certified wash. Data, not code, so a forker can open it and disagree with it

# THE COMPATIBILITY MATRIX, loaded from data/matrix.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
MATRIX_PATH = os.path.join(HERE, "data", "matrix.json")
GRADES = ("technical", "food_grade", "high_purity")
VERDICTS = ("accept", "clean_then_load", "refuse", "undetermined")
def load(path=MATRIX_PATH):
M = load()
WASH_LADDER = list(M["wash_ladder"])
WASH_RANK = {name: i for i, name in enumerate(WASH_LADDER)}
CLASS_OF = {}

src/prompt.pyprompt

assemble one call for all ten fields, with the matrix RENDERED into it from data/matrix.json rather than restated in prose — a lookup the model has never been shown is not a lookup

# Assemble the pre-load check prompt. One prompt per check sheet, all ten fields in it.

SYSTEM = (
def matrix_block(m=None):
MATRIX_TEXT = matrix_block()
def field_schema(fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key — plus the pure-code business-condition check downstream: a tank that is not clear to load, already loaded, is held

You change it to: the hold rule itself — this kit ships two values (not accept AND already loaded) and a real terminal weighs how much product is in the tank, whether it can still be diverted, and who has authority to release it. It is deliberately NOT the same function as the matrix lookup, so changing WHAT GETS HELD does not change WHAT IS COMPATIBLE

# Extract one pre-load check sheet's fields: segment, select, prompt, one model call, then a

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 4000
def load_fields():
def load_doc(check_id):
def documents():
def decide(values):
def correct_verdict(values):

evals/judge.pyjudge

score field accuracy, the four-way verdict AND its safety-relevant collapse (clear-to-load or not), and the hold flag, all separately, pure code

# Score a pre-load check run. PURE CODE -- gold is exact and the answer is one value per cell, so

def norm(v):
def equal(field, got, want):
def score(fields, records, golds):
def _matrix(rows, positive):
def score_verdicts(records, flags, golds):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2094 input and 488 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's pre-load check sheets are entirely synthetic (tools/build_corpus.py, seed 20260822): no real tank, terminal, operator, carrier, customer or shipment exists in the corpus, and nothing was fetched from anywhere. Every cargo is an ordinary generic chemical or oil name and no trade name is used. The only outbound traffic the kit makes is one chat-completion request per sheet to the configured provider, carrying the mapped sections of one sheet plus the fixed prompt and the shipped matrix. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser — which matters more here than usual, because this kit deliberately surfaces transport errors that carry a URL.

The experimentWe did not attack it — and three of four boundaries hold

The three boundaries that hold were confirmed by reading the code, not by a run: no code path authorises, releases or stops a load; the matrix lookup and the hold rule read only enums and cargo names, so the inspector's note cannot reach either; and the key and base URL are redacted out of any error before it is returned. The fourth is open. An indirect prompt injection needs a field an outside party authored, and this kit has exactly one — the inspector's note — which makes it the obvious place to attack and the reason the gap is named rather than glossed. Confirmed by reading the code, not by a run, on 2026-08-22 — no attack run was fired; see redteam.why.

The first three boundaries hold, confirmed by reading the code rather than by an attack run. The fourth is open and is named as open — an indirect prompt injection in the inspector's note is exactly the attack this kit's shape invites, and nothing here has tried it.

The result0 attack trials, and three of four boundaries checked here hold on evidence. The one that does not is the one this kit's own shape invites: the terminal inspector's note is free text a third party wrote, and whether an instruction hidden in it could move a safety verdict is unmeasured.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted ambiguity is a REGISTER mismatch — a relaxed note on a tank that must be refused — which measures whether a model runs the lookup when the prose points the other way. It does not measure whether a model obeys an instruction hidden in the same field. Those are different failures and only one of them is measured here.

HonestyWhat this does not prove

• Whether an indirect prompt injection in the inspector_notes field could move the verdict. No attack run exists.
• Whether the kit behaves safely against a hostile provider — a response body crafted to break the JSON extraction in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed document, and nothing beyond that was tested.
• Whether a real terminal's pre-load paperwork would carry anything sensitive this kit mishandles. The corpus has no personal data by construction; a real sheet carries a named inspector, a customer's product and a carrier's commercial history, and none of that path is exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_hold fires when the tank is NOT clear to load as it stands AND the product has already gone in — verdict != "accept" and load_status == "loaded". Both values come out of the same reply; the rule is run afterwards in pure code, over whatever the model returned, and never over gold. `undetermined` counts as not-clear, deliberately: product already in a tank whose prior cargo nobody recorded is precisely a hold. A reply missing either value returns None rather than False — an unknown is not a pass.

src/extract.py::compute(), called by extract() on every sheet, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by identical code. The compatibility lookup it reads lives in a SEPARATE module, src/matrix.py::decide(), which is also what the corpus generator wrote gold with and what src/prompt.py states to the model — one definition, three readers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT an authorisation, a refusal or a quarantine. Nothing here stops a load, releases a tank, notifies a terminal or writes to any system, and verdict is a proposal with reasoning attached rather than a decision anybody is bound by. A qualified person authorises the load.
• It is NOT a check on whether the extracted values are right. If the model misreads a cargo name or a certificate and then reasons correctly from the misreading, the tank is held (or not held) on a wrong fact and this rule cannot tell. The consistency diagnostic in evals/judge.py is the closest thing to that check, it is reported separately, and it is blind to exactly the same case.
• It is NOT a terminal's real quarantine policy. Not-clear-and-already-loaded is this kit's own simplification, invented for this corpus. No operator's procedure, industry code or regulatory rule was consulted, and none is reproduced. A real terminal weighs how much product is in the tank, whether it can still be diverted, what the receiving customer's own specification says, and who has authority to release it.
• It is NOT a judgement about the matrix. The flag reads the verdict the matrix produced; if the matrix is wrong, the flag is confidently wrong with it. The shipped matrix is illustrative and is not an authority.

WatchedWhat is watched, and why that one

The kit measures 18 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 11 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (failures 0, refusal_cells 0, thinking True) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

5 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 5 runs measured the same system — they differ on documents, extraction_cells, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 5 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/extract.py, on any change to data/matrix.json (which moves the verdict the flag reads), on any change to data/fields.json's allowed values for verdict or load_status, and on any corpus regeneration. Changing WHAT GETS HELD is a policy change and must be re-scored, even though it never changes what is compatible.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the compatibility matrix is a JSON file read with json, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per sheet — segment, select, prompt, one call, parse, compute — with no branch, no loop and no agent. Anything that looks like orchestration in a kit this size is a diagram of a straight line.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain — and this kit has a concrete bill for that, not a hypothetical one: the retry policy was ours, it was incomplete, and it cost two sheets of a paid run before anybody noticed. The JSON extraction from a fenced reply, the backoff policy, the section regex and the .env reader are all code somebody has to own.
• No framework means no framework's ecosystem — no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. What ships is what is in the repository.
• The provider abstraction covers exactly two wire formats. A third provider is one function and one dict entry, and until somebody writes it the kit runs on two shapes.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-cargo-compat on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 4 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

1 run not plotted. b000-cargo-compat-rules recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Every tank id, terminal name and inspector note is invented and every cargo is an ordinary generic chemical or oil name; no trade name, supplier, carrier or terminal operator is named. The compatibility matrix this kit ships (data/matrix.json) is ILLUSTRATIVE and is not an authority. It was written for this kit and reproduces no commercial, industry-body or proprietary chart, no carrier's or terminal operator's own prior-cargo list and no regulatory schedule. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-field exact match, light normalisation

Does the model's value for each field match gold, after trimming whitespace and punctuation? prior_cargo and two_back_cargo are the two legitimately-null fields — null where the sheet says the prior cargo is not recorded, or that the tank was recertified and carried nothing before the prior one — so a null there is a hit, not a miss, when gold agrees.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every call, and the same field-match logic the free floor is scored by — a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same values the sheet states; evals/check_labels.py asserts every non-nullable field is populated on every row, that prior_cargo's nullness agrees with the undetermined verdict in BOTH directions, that every cargo name is one the shipped matrix carries, and that each of the four hard cases clears its own floor, before any run is allowed to spend.

How tight can the band be? There is no tolerance band — it is exact match after trimming whitespace and punctuation, never a continuous score to round.

Cadence: Re-score on any change to data/matrix.json, src/prompt.py, data/fields.json or tools/build_corpus.py — the first three change what is asked, the fourth changes what is asked about. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe verdict, four-way and collapsed onto clear-to-load

Two questions at once. Four-way: did the run land on the same verdict the matrix produces — accept, clean_then_load, refuse or undetermined? And the one that is safety-relevant: OF EVERY TANK THAT MUST NOT BE LOADED AS IT STANDS, HOW MANY DID THE RUN STOP? not accept is the positive class, and the expensive direction — a tank the matrix would have blocked that the run cleared — is counted and named on its own as unsafe_release, never averaged into an accuracy figure where it disappears.

MethodHow the test was run

evals/judge.py::score_verdicts, in-process, no key and no model.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's verdict is re-derived inside the grader by the same lookup the kit publishes — src/matrix.py::decide(), reading data/matrix.json — so the truth this matrix grades against can never be a separately-typed label that drifted from the rule.

How tight can the band be? No threshold — the verdict is one of four allowed values, and a reply returning none of them is counted as unanswered rather than folded into a correct cell. On a pre-load check that distinction is the whole difference between a tank nobody cleared and a tank somebody cleared.

Cadence: Re-run whenever data/matrix.json or src/matrix.py changes, whenever the corpus is regenerated, and on any provider or model change.

The decisionWhen to reach for it

In one lineneeds_hold against the same rule run over gold

Does the pure-code hold decision — not clear-to-load AND already loaded — land on the same tanks it would land on if both fields had been read perfectly? It is a BUSINESS CONDITION, so unlike a self-consistency check it genuinely needs labels, and saying so is half of what makes the number believable.

MethodHow the test was run

evals/judge.py::score_verdicts, in-process, no key and no model.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute(), the same function the run uses, applied to GOLD's verdict and load_status. One rule, two inputs, so a change to the rule moves both sides together and the grader cannot silently grade an old policy.

How tight can the band be? No threshold — two values and a comparison.

Cadence: Re-run whenever compute() changes. Changing WHAT GETS HELD is a policy change and must be re-scored, even though it never changes what is compatible.

The decisionWhen to reach for it