The business caseThe problem this solves

A pack of paperwork arrives with a delivered item and somebody at goods-in has to decide whether to let it into stock. Every line of it -- the certificate of conformity, the test report, the packing list, the custody chain, sometimes a letter from the maker appointing the distributor -- was written by parties OUTSIDE the organisation receiving it. The question is not whether it looks right; it is whether the pack's own claims survive contact with what the buyer already knows, and where they do not, which disagreement decides. That is fifteen separate checks against three registers, four date subtractions and a chain equality, per pack, and the answer has to name which one decided or the record cannot be confirmed by anybody. The reviewer's manual pass over one document pack against the buyer's registers and the rulebook: reading the certificate, the test report, the packing list and the chain; looking the part, the specification revision and the supplier up; subtracting dates; and writing the finding down. It replaces the READING and the LOOKUP. It does not replace the confirmation, and it writes to nothing.

The inputThe actual document packs

The corpus is 60 document packs, 0.06 MB (txt 60). The cases are the ones that make this decision hard, and none of them can be scraped -- real receiving packs name real suppliers and travel with real accusations. The four that matter: a certificate calling up a specification revision ISSUED AFTER the stated manufacture date (a revision that did not exist yet cannot have been built to); a custody chain that hands the lot to one party and receives it from another, with nothing recording the step between; an unapproved supplier who holds the maker's letter for THIS part, in date on the day of receipt -- a normal source that a register check alone condemns; and a pack that refers to a raw-material certificate not in the folder, where nothing disagrees with anything else. That last one is the reason INSUFFICIENT_EVIDENCE exists as its own verdict: it is not suspicion, it is not enough paper, and calling it either clean or suspect is wrong in a different direction.

Goods-in note -- PA-0001
Written up 18 Jun 2026. Consignment against PO-88011 from Stonebridge Surplus, who I could not find a supplier code for.

The order was raised for part DF-9034-01, 50 off.
What came in is part DF-9034-01, 50 pieces, and the pieces carry date code 2547.
There is a certificate of conformity in the pack from Stonebridge Surplus. It gives the lot as L-25-1072, says it was made by Dunmore Fastener Works on 20 November 2025, and calls up SPEC-DF-330 revision D. It covers 50 pieces.
It is signed by D. Prakash, Certification Officer.
Test report TR-11037 is with it, for lot L-25-1072, dated 22 November 2025.
Traceability as the supplier states it: Dunmore Fastener Works consigned it to Stonebridge Surplus on 7 December 2025 (TN-40013); Stonebridge Surplus then to Halden Aerostructures Ltd on 17 December 2025 (DN-90007).
Documents physically in the folder: purchase order, certificate of conformity, test report, packing list, traceability chain, authorisation letter.
They have also put in a letter from Dunmore Fastener Works appointing Stonebridge Surplus for DF-9034-01, which runs from 7 April 2025 until 2 February 2027.

Filed for engineering review. No disposition taken at goods-in.

The outcomeWhat a good result looks like

One authenticity risk record a person confirms instead of a folder somebody reads: one verdict from a closed four-value vocabulary, the SET of every indicator that fired including the informational one, the ones that actually decided it, one sentence per indicator naming the two things that disagree, and checks_not_run -- every rule that could not run and why, because a check that cannot run does not pass.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Packs that arrive as a CSV manifest, a labelled QMS export or a document-by-document supplier packet — the free rules floor alone
  36 of 36 all-correct on those three formats, identical to the paid arm, for $0.00. Every register lookup, every date subtraction and the whole tier precedence are the engine, and the engine is free. 13 of the 16 planted cases are a dead tie between the arms.
• Packs written as free prose by whoever was on the receiving bay, where the custody chain and the authorisation letter arrive as sentences — the model -- but read the margin before you buy it
  24 of 24 against the floor's 20 of 24 on inspection_note, and the four packs it wins are the two error directions in equal measure: two false clears (a provably impossible chain into stock) and two false alarms (a legitimate distributor accused). Those are the expensive mistakes, not the cheap ones.

And where nothing here is good enough:

• Anything where the answer must establish that the ITEM is genuine — neither arm -- this kit cannot answer that question
  Every indicator here is a statement about DOCUMENTS: do the pack's claims agree with each other and with the buyer's registers. A competently forged pack that agrees with itself scores AUTHENTIC, and a genuine part with sloppy paperwork does not.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-31. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-31 — r001-part-authenticity. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — ⚠︎ PARTIALLY EVIDENCED, AND THE REST IS NOT MEASURED. What is on disk and checkable by anybody with no key: the 60 packs, the registers, the rulebook, the answer key, all four result files and the board. python3 -m evals.check_labels grades the key by a second implementation and prints KEY CLEAN at 60 of 60. python3 -m tools.build_corpus rebuilds the corpus from seed 20260831 -- VERIFIED BYTE-IDENTICAL across four PYTHONHASHSEED values (0, 7, 12345, 99991) and against the committed files, in throwaway copies, on 2026-08-31. The rules floor and the stub arm re-run to their committed scores for $0.00. What is NOT verified from cold: the paid arm, which needs a key and would re-buy 60 calls.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one document pack end to end -- one pack, one model call, including provider-side reasoning tokens, on a shared connection. 94.3 pct of the output tokens on this run never reach text, so the wait is mostly the provider thinking, not the answer arriving.

Current processWhat it replaces

The reviewer's manual pass over one document pack against the buyer's registers and the rulebook: reading the certificate, the test report, the packing list and the chain; looking the part, the specification revision and the supplier up; subtracting dates; and writing the finding down. It replaces the READING and the LOOKUP. It does not replace the confirmation, and it writes to nothing.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one authenticity risk record is

No train/test split, because nothing is trained or tuned, and no segmentation, because one pack goes whole into one prompt. All 60 packs are scored on every arm and every published figure is over all 60 or over a named sub-denominator (49 planted, 11 clean, 5 trap, 36 risk, 24 non-risk, 56 indicator labels).

SetupWhat the setup figure measured

THERE IS NO INDEX. There is nothing to retrieve from: one pack goes whole into one prompt together with the rulebook and the buyer's registers, whole. No embeddings, no chunking, no vector store, no top-k. EMBED_MODEL is unused. The registers are sent un-sliced ON PURPOSE -- slicing them on anything the model read would hand it the answer, because a part that is absent from the catalogue is itself a finding.

LicenceLicence

MIT -- the text is in the repository's LICENSE-PUBLIC, not in its root LICENSE, which governs the repository itself and grants a reader nothing. The kit, its code and this generated corpus are MIT; the repository is not, and it is private. data/SOURCES.md states the distinction and names the file. Every buyer, maker, distributor, part, specification, lot, certificate and register entry here is invented and so is the rulebook, so there is no third-party data in this kit to licence and nothing to clear.

Bring your ownBring your own document packs

Drop your own packs as .txt into data/corpus/, add one row per pack to data/packs.json (id, received_at, format), replace data/registers.json with your own catalogue, specification register and approved-supplier list, and either label data/gold.jsonl yourself or let src/rules.py derive it from a reading you supply. data/rulebook.json is where your own indicators, tiers, tolerances and precedence go -- src/rules.py hard-codes none of it. Then run evals/check_labels.py first: it will refuse a key whose identifiers do not appear in the packs' own text.

⚠︎ And what stops being true when you do: ⚠︎ A REAL DOCUMENT PACK NAMES A REAL SUPPLIER AND, IF THE ANSWER IS A RISK VERDICT, AMOUNTS TO AN ACCUSATION ABOUT THEM. It also carries the maker's certificate, lot traceability and often commercial terms, all of which are somebody else's confidential documents held under a purchase contract, not the buyer's to send anywhere. Sending one to a hosted provider sends all of that. Run this against packs you are permitted to disclose, or against a local model. And note the second boundary: this kit produces a RECORD, not a disposition. It does not quarantine, reject, contact a supplier or write to any register, and there is no endpoint that does.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 18,440 characters assembled, 7,811 are instructions — 42% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The exact six parts sent for PA-0001, the first pack of run r001-part-authenticity, replayed from the kit's own src/prompt.py against the committed pack and registers. Character counts total 18,440 and match the run record's prompt_chars_total exactly. Only receipt and pack vary between calls.

Full promptThe prompt in full

You are the receiving-inspection desk of Halden Aerostructures Ltd, working ONE DOCUMENT PACK: the paperwork that arrived
with a delivered part. Your output is the AUTHENTICITY RISK RECORD an engineer confirms -- so that
the engineer confirms a record instead of reading a folder.

How to read it:

- THE PACK IS A CLAIM. Every date, every party name, every specification revision and every quantity
  in it was written by somebody outside this organisation. The REGISTERS below are what this buyer
  knows: which maker makes each part, when each specification revision was ISSUED, and who is
  approved to supply. Where the pack and a register disagree, the register is the record and the
  pack is the claim.
- Read the pack's own fields off the pack whatever shape it arrives in -- an export with labelled
  fields, a one-row manifest, a document-by-document transcription, or an inspector's written note.
  Read a date written any way at all into YYYY-MM-DD.
- Read the CUSTODY CHAIN as an ordered list of hand-offs, each one from a party to a party. It joins
  up when every link receives the lot from the party the previous link handed it to.
- Read WHICH DOCUMENTS ARE PHYSICALLY IN THE PACK, and separately WHICH DOCUMENTS THE PACK REFERS
  TO. A certificate that cites a raw material certificate has not produced one.
- Then apply the rulebook as written, including its arithmetic: the revision issue date against the
  manufacture date, the date code decoded under the maker's own scheme against the manufacture date
  and the tolerance, the letter's validity dates against the receipt date, the quantities, the lot
  identifiers.
- Name EVERY indicator that fired, by its id, including the informational one. Then choose exactly
  one verdict, by the order the rulebook gives.
- A check that cannot be run because the document it needs is not in the pack DOES NOT PASS. Say so
  by naming required_document_missing, and do not let the absence become a clean bill.

THE FOUR VERDICTS -- exactly one per pack, in the order they are tried:

  COUNTERFEIT_INDICATED
    At least one counterfeit-tier indicator fired: the claimed custody chain does not join up,
    it does not start at the maker of record, the certificate cites a specification revision
    issued after the date the part was made, or the supplying party has no authority to supply
    it. These are the findings a genuine chain cannot produce.
    Cost of getting it wrong: Missed, a part whose paperwork is provably impossible goes into stock with a clean
    receiving record, and every downstream record inherits it. Raised wrongly, a legitimate
    consignment is quarantined and a supplier relationship is accused -- which is why the
    indicator that raised it is named on the record and the verdict never stands on its own.

  SUSPECT
    At least one suspect-tier indicator fired and no counterfeit-tier indicator did. Something
    in the pack disagrees with something else in it, or with the buyer's own registers, in a
    way a clerical error can also produce: a date code from a different production week, a
    test report for a different lot, more pieces delivered than the certificate covers, an
    unsigned certificate.
    Cost of getting it wrong: Collapsed into AUTHENTIC it is a disagreement nobody looked at. Escalated to
    COUNTERFEIT_INDICATED it turns a transcription slip into an accusation, and the next real
    one is read as another false alarm.

  INSUFFICIENT_EVIDENCE
    No risk indicator fired, and the pack does not carry enough to say it is clean: a required
    document is absent, or the paperwork is internally consistent and refers to a document
    that is not here. NOTHING DISAGREES. There is simply not enough of it.
    Cost of getting it wrong: This is the verdict a two-way checker cannot express, and both ways of losing it are
    wrong. Called AUTHENTIC, a chain is certified on a document nobody has read. Called
    SUSPECT, a supplier is accused of something the pack does not show, and the answer -- send
    the missing document -- is never asked for.

  AUTHENTIC
    Every required document is in the pack, every check the rulebook could run has run, and
    none of them raised a risk indicator. Informational indicators may still be named: a
    supplier outside the approved register who holds the maker's authorisation letter,
    covering this part, in date on the day of receipt, is a normal source and is recorded as
    one.
    Cost of getting it wrong: Given to a pack whose paperwork is impossible, it is the whole failure of the job.
    Withheld from a pack that earns it, an authorised distributor is treated as a counterfeit
    risk on the strength of an out-of-date register -- the false positive a naive rulebook
    produces, and the one this corpus plants on purpose.

Reply with JSON and nothing else, in the shape given at the end.

THE RULEBOOK, as this desk applies it.

IT IS ILLUSTRATIVE. It was written for this kit and reproduces no purchasing standard, quality clause or
regulator's guidance.

DOCUMENTS THE PACK MUST CONTAIN
  purchase_order, certificate_of_conformity, test_report, packing_list, traceability_chain
  A required document absent from the pack does not make the part suspect. It makes the pack
  unjudgeable on every rule that needs it -- which is why those rules record nothing rather
  than passing, and why INSUFFICIENT_EVIDENCE exists as a verdict of its own.
  Documents that may also appear: authorisation_letter, raw_material_certificate, inspection_report, export_declaration

THRESHOLDS
  date code tolerance   6 weeks either side of the stated manufacture date
  A maker's date code is a production week, not a ship date. Codes are stamped ahead of and
  behind the paperwork's manufacture date in normal production, so the tolerance is wide on
  purpose: outside it, the code and the lot are describing different production, and that is
  the finding.

INDICATORS -- COUNTERFEIT tier
  chain_break
    The claimed custody chain does not join up
    A link in the traceability chain names a transferor that the previous link never named as
    holder. The lot left one party's custody and arrived in another's with nothing recording
    the step between.
    needs in the pack: traceability_chain
  chain_origin_not_maker
    The chain does not start at the maker of record
    The first link of the traceability chain originates with a party that is not the maker the
    buyer's part catalogue holds for this part number.
    needs in the pack: traceability_chain
  revision_postdates_manufacture
    The certificate cites a specification revision that did not exist yet
    The specification revision the certificate of conformity cites was issued AFTER the
    manufacture date the same certificate states. A part cannot have been built to a revision
    that had not been published when it was built.
    needs in the pack: certificate_of_conformity
    The revision issue dates come from the buyer's specification register, never from the
    certificate. A certificate that supplies its own revision dates is supplying its own
    alibi.
  unauthorised_source
    The supplier has no authority to supply this part
    The supplying party is not on the buyer's approved-supplier register (absent, suspended,
    or its approval had lapsed on the receipt date) AND the pack carries no authorisation
    letter from the maker of record that names this distributor, covers this part number, and
    is in date on the receipt date.
    The letter must be checked, not merely present: a letter for a different part number, or
    one that expired before the goods were supplied, authorises nothing.

INDICATORS -- SUSPECT tier
  date_code_outside_window
    The date code and the stated manufacture date describe different production
    The date code marked on the goods decodes, under the maker's own scheme, to a production
    week more than the rulebook's tolerance away from the manufacture date the certificate
    states.
    needs in the pack: packing_list, certificate_of_conformity
  lot_mismatch_test_report
    The test report is for a different lot
    The lot identifier on the test report is not the lot identifier on the certificate of
    conformity.
    needs in the pack: test_report, certificate_of_conformity
  part_number_mismatch
    What was delivered is not what was ordered
    The part number on the delivery paperwork is not the part number on the purchase order.
    needs in the pack: purchase_order, packing_list
  quantity_exceeds_certified
    More was delivered than the certificate covers
    The quantity on the packing list is greater than the quantity the certificate of
    conformity certifies.
    needs in the pack: packing_list, certificate_of_conformity
  certificate_unsigned
    The certificate names no authorised signatory
    The certificate of conformity carries no signatory -- no name, no role, nobody who is
    asserting anything.
    needs in the pack: certificate_of_conformity
  test_report_predates_manufacture
    The lot was tested before it was made
    The test report is dated earlier than the manufacture date the certificate states.
    needs in the pack: test_report, certificate_of_conformity

INDICATORS -- EVIDENCE tier
  required_document_missing
    A document the pack must contain is not in it
    One of the rulebook's required documents is absent from the pack. Every rule that needs it
    records nothing -- it does not pass.
  referenced_document_absent
    The pack refers to a document that is not in the pack
    A document inside the pack names another document as supporting it -- a raw material
    certificate, an inspection report, an export declaration -- and that document is not among
    the ones present.
  specification_revision_unknown
    The cited specification revision is not in the buyer's register
    The certificate cites a specification or a revision the buyer's specification register
    does not hold, so the revision-date check cannot be run at all.
    needs in the pack: certificate_of_conformity
  part_not_in_catalogue
    The part number is not in the buyer's catalogue
    The delivered part number is not in the buyer's part catalogue, so there is no maker of
    record to check the chain's origin against and no specification to check the revision
    against.

INDICATORS -- INFORMATIONAL tier
  supplier_not_on_approved_list
    The supplier is not on the approved-supplier register
    The supplying party is absent from the register, suspended on it, or its approval had
    lapsed by the receipt date. Recorded on every pack where it is true, and it decides
    nothing on its own.

THE ORDER THE VERDICTS ARE TRIED IN
  COUNTERFEIT_INDICATED -> SUSPECT -> INSUFFICIENT_EVIDENCE -> AUTHENTIC
  Fixed order, applied top down over the indicators that actually fired. A finding you have
  outranks information you lack, which is why INSUFFICIENT_EVIDENCE sits BELOW both risk
  verdicts and not above them: a pack missing its test report and carrying a broken custody
  chain is a counterfeit indication, not an evidence gap. The informational tier never decides
  -- it is named on the record and skipped by this order.

THE BUYER'S REGISTERS -- Halden Aerostructures Ltd's own records, as at 2026-08-20. THIS is what the buyer knows; the pack is what it was told.

PART CATALOGUE -- who makes each part, and to which specification
  part           maker of record            specification  date code scheme
  BR-7712-11     Braeling Forgeworks        SPEC-BR-204    YYWW
  BR-7712-14     Braeling Forgeworks        SPEC-BR-204    YYWW
  CP-2260-03     Colthart Precision         SPEC-CP-071    YYWW
  CP-2260-08     Colthart Precision         SPEC-CP-071    YYWW
  CP-2260-12     Colthart Precision         SPEC-CP-071    YYWW
  DF-9034-01     Dunmore Fastener Works     SPEC-DF-330    YYWW
  DF-9034-06     Dunmore Fastener Works     SPEC-DF-330    YYWW
  ES-5521-04     Eastvale Seals             SPEC-ES-045    YYWW
  ES-5521-09     Eastvale Seals             SPEC-ES-045    YYWW
  FA-1307-02     Fenwick Alloys             SPEC-FA-512    YYWW
  FA-1307-07     Fenwick Alloys             SPEC-FA-512    YYWW
  HX-4180-02     Arden Components           SPEC-HX-118    YYWW
  HX-4180-05     Arden Components           SPEC-HX-118    YYWW
  HX-4180-09     Arden Components           SPEC-HX-118    YYWW

SPECIFICATION REGISTER -- the day each revision was ISSUED
  SPEC-BR-204    Forged bracket, primary structure  A 2017-11-30, B 2020-05-18, C 2022-10-04, D 2024-08-12, E 2026-02-09
  SPEC-CP-071    Precision bushing, close tolerance A 2019-04-08, B 2021-12-06, C 2024-03-25, D 2026-05-11
  SPEC-DF-330    Threaded fastener, high temperature A 2016-07-19, B 2019-09-02, C 2022-01-17, D 2025-06-30
  SPEC-ES-045    Elastomeric seal, fuel contact     A 2018-03-05, B 2020-11-23, C 2023-02-13, D 2025-10-06
  SPEC-FA-512    Alloy billet, certified stock      A 2017-05-22, B 2021-08-16, C 2024-01-29, D 2026-03-16
  SPEC-HX-118    Hydraulic manifold block, machined A 2018-09-14, B 2021-02-11, C 2023-06-02, D 2025-01-20, E 2026-04-27

APPROVED-SUPPLIER REGISTER -- who may supply, and until when
  code       supplier                   status      approved until
  SUP-3021   Northgate Distribution     approved    2027-03-31
  SUP-3044   Larkfield Supply Group     approved    2027-09-30
  SUP-3070   Vantry Aerospace Supply    approved    2028-01-31
  SUP-3112   Ormsby Industrial          approved    2027-06-30
  SUP-3155   Calderwood Trading         approved    2027-12-31
  SUP-3188   Penhale Components Ltd     approved    2027-05-31
  SUP-3203   Redwyn Parts Co.           suspended   2027-08-31
  SUP-3241   Tarnbrook Supply           approved    2026-03-31
  A party that appears on the paperwork and in no row above is NOT on the register.
  Only the party that supplied the buyer -- the last hand-off in the chain -- is checked
  against this register. An intermediate trader earlier in the chain is not.

THE RECEIVING RECORD -- this consignment as the buyer's own goods-in queue holds it.

  pack reference        PA-0001
  goods received on     2026-06-18

Every validity test that is not against the certificate is against the receipt date above.

THE DOCUMENT PACK, verbatim:

Goods-in note -- PA-0001
Written up 18 Jun 2026. Consignment against PO-88011 from Stonebridge Surplus, who I could not find a supplier code for.

The order was raised for part DF-9034-01, 50 off.
What came in is part DF-9034-01, 50 pieces, and the pieces carry date code 2547.
There is a certificate of conformity in the pack from Stonebridge Surplus. It gives the lot as L-25-1072, says it was made by Dunmore Fastener Works on 20 November 2025, and calls up SPEC-DF-330 revision D. It covers 50 pieces.
It is signed by D. Prakash, Certification Officer.
Test report TR-11037 is with it, for lot L-25-1072, dated 22 November 2025.
Traceability as the supplier states it: Dunmore Fastener Works consigned it to Stonebridge Surplus on 7 December 2025 (TN-40013); Stonebridge Surplus then to Halden Aerostructures Ltd on 17 December 2025 (DN-90007).
Documents physically in the folder: purchase order, certificate of conformity, test report, packing list, traceability chain, authorisation letter.
They have also put in a letter from Dunmore Fastener Works appointing Stonebridge Surplus for DF-9034-01, which runs from 7 April 2025 until 2 February 2027.

Filed for engineering review. No disposition taken at goods-in.


Reply with JSON and nothing else, exactly this shape:

{"reading": {
   "part_number": "<the part actually delivered, as the pack gives it>",
   "po_part_number": "<the part the purchase order was raised for, or null if there is no purchase order>",
   "purchase_order": "<the order reference, or null>",
   "supplier_code": "<the supplier's code on the buyer's systems, or null if the pack gives none>",
   "supplier_name": "<the party that supplied the buyer -- the LAST hand-off in the chain>",
   "maker_named": "<the maker the certificate names, or null>",
   "lot_id": "<the lot the certificate certifies, or null>",
   "manufacture_date": "<YYYY-MM-DD, the date the certificate states the lot was made, or null>",
   "specification": "<the specification the certificate calls up, e.g. SPEC-HX-118, or null>",
   "specification_revision": "<the revision letter, or null>",
   "certified_quantity": <integer the certificate covers, or null>,
   "delivered_quantity": <integer the packing list states, or null>,
   "date_code": "<the code marked on the pieces, or null>",
   "certificate_signatory": "<the name and role signing the certificate, or null if it is unsigned>",
   "test_report_lot": "<the lot the test report is for, or null>",
   "test_report_date": "<YYYY-MM-DD, or null>",
   "documents_present": ["<one of: purchase_order | certificate_of_conformity | test_report | packing_list | traceability_chain | authorisation_letter | raw_material_certificate | inspection_report | export_declaration>"],
   "references": [{"kind": "<a document kind from the list above>", "ref": "<its reference, or null>"}],
   "chain": [{"from": "<party>", "to": "<party>", "date": "<YYYY-MM-DD>", "ref": "<transfer reference or null>"}],
   "authorisation_letter": {"maker": "<issuing maker>", "distributor": "<party appointed>",
                            "parts": ["<part numbers covered>"], "valid_from": "<YYYY-MM-DD>",
                            "valid_to": "<YYYY-MM-DD>"}},
 "verdict": "COUNTERFEIT_INDICATED" | "SUSPECT" | "INSUFFICIENT_EVIDENCE" | "AUTHENTIC",
 "indicators": ["<every indicator id that fired, from: chain_break, chain_origin_not_maker, revision_postdates_manufacture, unauthorised_source, date_code_outside_window, lot_mismatch_test_report, part_number_mismatch, quantity_exceeds_certified, certificate_unsigned, test_report_predates_manufacture, required_document_missing, referenced_document_absent, specification_revision_unknown, part_not_in_catalogue, supplier_not_on_approved_list>"],
 "why": "<one sentence naming the indicator that decided the verdict, or saying that none fired>"}

`documents_present` lists only documents that are IN the pack. `references` lists documents the pack
POINTS AT; a reference whose kind is not in documents_present is the whole point of the field.
`authorisation_letter` is null when the pack carries none. `indicators` is an empty list when
nothing fired.

Raw responseThe raw response

{"reading": {
  "part_number": "DF-9034-01",
  "po_part_number": "DF-9034-01",
  "purchase_order": "PO-88011",
  "supplier_code": null,
  "supplier_name": "Stonebridge Surplus",
  "maker_named": "Dunmore Fastener Works",
  "lot_id": "L-25-1072",
  "manufacture_date": "2025-11-20",
  "specification": "SPEC-DF-330",
  "specification_revision": "D",
  "certified_quantity": 50,
  "delivered_quantity": 50,
  "date_code": "2547",
  "certificate_signatory": "D. Prakash, Certification Officer",
  "test_report_lot": "L-25-1072",
  "test_report_date": "2025-11-22",
  "documents_present": [
    "purchase_order",
    "certificate_of_conformity",
    "test_report",
    "packing_list",
    "traceability_chain",
    "authorisation_letter"
  ],
  "references": [],
  "chain": [
    {"from": "Dunmore Fastener Works", "to": "Stonebridge Surplus", "date": "2025-12-07", "ref": "TN-40013"},
    {"from": "Stonebridge Surplus", "to": "Halden Aerostructures Ltd", "date": "2025-12-17", "ref": "DN-90007"}
  ],
  "authorisation_letter": {
    "maker": "Dunmore Fastener Works",
    "distributor": "Stonebridge Surplus",
    "parts": ["DF-9034-01"],
    "valid_from": "2025-04-07",
    "valid_to": "2027-02-02"
  }
},
"verdict": "AUTHENTIC",
"indicators": ["supplier_not_on_approved_list"],
"why": "No risk-tier indicator fired; the only indicator recorded is the informational supplier_not_on_approved_list, which is covered by a valid authorisation letter from the maker naming Stonebridge Surplus for this part."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch a supplier's parts paperwork that doesn't add up — 60 document packs. One model answered, and every answer was then graded Two different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

No model grades anything. Every metric is exact match against data/gold.jsonl: the verdict as a string from a closed four-value vocabulary, the indicators as a SET of ids compared with set equality. Precision and recall are computed over the key's 56 indicator labels, micro and macro, with both error DIRECTIONS reported separately because they cost different things. The graders are pure Python and re-run for $0.00 against the committed result files.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Run it twiceThe same set, run again

The same 60 replies scored twice, and the two columns are IDENTICAL -- 100.0 per cent and 100.0 per cent, field for field, with recheck_overrides 0 and recheck_unrecheckable 0. The rule engine re-derived every indicator and every verdict from the model's own reading and disagreed with it on nothing.

record-all-correct over the 60 document packs -- the verdict and the indicator set both right on one pack — They are not two samples of anything. They are one sample scored by two rules, so averaging them would report a spread that does not exist. ⚠︎ AND HERE THE PAIR IS EVEN LESS INFORMATIVE THAN USUAL: identical columns with 0 overrides mean the guardrail was never exercised. A control that never fires has not been shown to work.

Grading costWhat it costs

Every dollar on this page is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a card, not a bill anybody paid. The measured tokens are 281,959 in and 414,853 out over 60 calls.

Rates checked 2026-08-27. The provider that actually ran every call here is kept off this page per the series rule. The real spend, its own dated card and the peak/off-peak split are recorded in results/eval-r001-part-authenticity.json.

Grading unitWhat the grading figure prices

Scoring is pure Python over committed JSON: 60 packs in well under a second, no key, no network. The ARM costs money -- $0.294608 for the 60 calls of r001-part-authenticity -- and the FLOOR costs nothing at all (results/eval-b000-part-authenticity-rules.json records usd null and wall_seconds 0.0).

The gradersTwo ways to grade

TWO FLOORS, AND THE SECOND IS THE ONE THAT MATTERS. The NULL baseline is answering the same verdict on every pack with no reading at all: the best single guess is COUNTERFEIT_INDICATED or SUSPECT at 18 of 60 verdicts (30.0 pct), and on the record grader -- verdict AND indicator set -- always-AUTHENTIC-with-no-indicators scores 11 of 60 (18.3 pct) because only the 11 clean packs have an empty set. That is the number a coin has. ⚑ THE REAL FLOOR IS b000-part-authenticity-rules: a keyless reader in front of the SAME rule engine, 56 of 60 (93.3 pct) for $0.00, with the indicator set exactly right on the same 56 packs. It is four packs behind the paid arm and that is the entire commercial question this kit poses. ⚠︎ AND THE ANSWER IS UNFLATTERING. The four-pack gap is 36 of 36 for BOTH arms on the three structured formats and 20 of 24 against 24 of 24 on the one renderer that writes prose. Paired over the 60 packs there are only 4 discordant packs, all favouring the model: McNemar exact two-sided p = 0.125 (one-sided 0.0625), so the gap is NOT distinguishable from chance at the conventional level and must not be quoted as one. And it is not even a reading-difficulty result: the generator writes the custody chain and the authorisation letter with exactly ONE sentence template each, and two regexes for those forms lift the free arm to 60 of 60.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-27. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingGrading is free; the bill is the 60 calls that answered.

The scorer, the label gate, the rules floor, the corpus rebuild and the stub arm are pure code and cost $0.00 to run against any result set, with no key and no network. The only priced run in this kit is r001-part-authenticity -- 60 calls, one per pack. The adversarial arm was never fired, so there is no second paid run to add. Real spend on the running provider: $0.294608.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, AND MOST OF THEM ARE NOT THE ANSWER. 391,125 of 414,853 output tokens (94.3 pct) are provider-side reasoning left at the tier's default and re-rolled per call, and the output side is 89.8 pct of the projected bill. The visible answer is a JSON object of a few hundred tokens: the PA-0001 reply is ~1,300 characters and was billed 23,412 output tokens, 22,925 of them reasoning.
• THE STABLE PREFIX, WHICH WORKS IN YOUR FAVOUR. The role, the rulebook and the registers are byte-identical across all 60 calls and 76.1 pct of the prompt's characters. MEASURED: 193,536 of 281,959 input tokens came back as cache hits, 68.6 pct. The projection cards below have no cache tier and price all of it at the full input rate.
• THE PACK ITSELF IS THE SMALLEST PART OF THE PROMPT -- 6.8 pct of characters, 1,250 of 18,440 on PA-0001. The bill is the rulebook, the registers, the schema and the reasoning, not the document. That inverts with real packs, which are not 1014 bytes.

Your volumeWhat it costs at your volume

Linear in packs and flat in everything else. Ten times the packs is ten times the calls at the same per-pack cost: there is no index to rebuild, nothing to embed and no state carried between calls. ⚠︎ WHAT IS NOT FLAT IS THE PROMPT. The registers go in WHOLE on every call by design, so a buyer with a thousand-part catalogue pays for that catalogue sixty times over -- the 2,796-character register block here is 15.2 pct of the prompt against the pack's 6.8 pct, and that ratio only worsens. The cache absorbs some of it (68.6 pct of input tokens were prefix hits) and only on a provider with a cached-input tier.

Model choiceWhy this model

It is the tier the estate runs, so the figure compares with every sibling kit. ⚠︎ AND THE FINDING HERE IS NOT ABOUT THE MODEL AT ALL. It answered every pack correctly, which means this corpus cannot tell it apart from a perfect arm -- or, on the evidence of the prose experiment, from free code. Nothing on this page supports 'this tier reads authenticity paperwork'; what it supports is 'this corpus has run out of difficulty'.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

the whole 60-pack scored run (r001-part-authenticity, the fast tier): 281,959 input tokens (193,536 of them prefix cache hits) and 414,853 output (391,125 of them provider-side reasoning). ⚠︎ usd_actually_paid is null ON THIS PAGE by the series rule, not because it is unknown: the real bill is $0.294608, every call landed in the off-peak tariff, and it is recorded per call with its tariff in the run record. The totals reconcile to the per-call records exactly, so this covers all 60 calls with none lost.

The codeEvery file, and what it is for

14 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Eight of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/prompt.pyprompt assembly

Six parts in a fixed order: the role and the rules of reading; data/rulebook.json rendered as text -- every indicator, its tier, what it means, the documents it needs and the precedence; the buyer's three registers WHOLE; the receiving record; the pack verbatim; the JSON schema. The four verdict definitions are rendered FROM VERDICT_MEANINGS rather than typed beside it, and the module asserts at import that every verdict and every indicator name appears in both the rulebook text and the schema.

# The assembled prompt, in one place, in the order it is sent.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULES = json.load(open(os.path.join(HERE, "data", "rulebook.json"), encoding="utf-8"))
VERDICTS = ("COUNTERFEIT_INDICATED", "SUSPECT", "INSUFFICIENT_EVIDENCE", "AUTHENTIC")
VERDICT_MEANINGS = {
INDICATORS = R.INDICATORS
def _verdict_block():
def _wrap(text, indent, width=94):
SYSTEM = """You are the receiving-inspection desk of %s, working ONE DOCUMENT PACK: the paperwork that arrived
SCHEMA = """Reply with JSON and nothing else, exactly this shape:

src/prompt.pythe verdict vocabulary

VERDICTS and VERDICT_MEANINGS, both module-level, both literals. Four verdicts, each with what it means and what it costs in each direction. This is the one copy: the prompt renders it, the scorer grades against it and the site reads it by AST.

# The assembled prompt, in one place, in the order it is sent.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULES = json.load(open(os.path.join(HERE, "data", "rulebook.json"), encoding="utf-8"))
VERDICTS = ("COUNTERFEIT_INDICATED", "SUSPECT", "INSUFFICIENT_EVIDENCE", "AUTHENTIC")
VERDICT_MEANINGS = {
INDICATORS = R.INDICATORS
def _verdict_block():
def _wrap(text, indent, width=94):
SYSTEM = """You are the receiving-inspection desk of %s, working ONE DOCUMENT PACK: the paperwork that arrived
SCHEMA = """Reply with JSON and nothing else, exactly this shape:

data/rulebook.jsonthe rulebook — a swap seam

Fifteen indicators in four tiers -- counterfeit, suspect, evidence, informational -- each with its meaning, the documents it needs and its precedence. Illustrative: it restates no purchasing standard, quality clause or regulator's guidance.

You change it to: The date-code window (six weeks here) and every other numeric tolerance, as data.

# 

src/rules.pythe rule engine

Reads the rulebook and hard-codes none of it. Given a reading, it re-derives every indicator by lookup and arithmetic, applies the four-tier precedence and writes the verdict, the deciding indicators, one finding sentence each and checks_not_run. It asserts its own branch order against the published precedence at import, so an engine whose branches have drifted from the order it prints cannot start.

# The indicator engine: the rulebook as arithmetic and lookups. Pure code, no model, no key.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULES = json.load(open(os.path.join(HERE, "data", "rulebook.json"), encoding="utf-8"))
BY_ID = {r["id"]: r for r in RULES["rules"]}
INDICATORS = tuple(r["id"] for r in RULES["rules"])
TIERS = {k: tuple(v) for k, v in RULES["tiers"].items()}
TIER_OF = {i: t for t, ids in TIERS.items() for i in ids}
VERDICTS = tuple(RULES["precedence"])
RISK_VERDICTS = tuple(v for v in VERDICTS if RULES["verdicts"][v]["risk"])
REQUIRED_DOCS = tuple(RULES["documents"]["required"])

src/registers.pythe registers

The buyer's part catalogue (14 parts, 6 makers), specification register (6 specifications, 26 dated revisions) and approved-supplier list (8 rows). Every lookup the engine does is against this file, never against a sentence in the pack.

# The buyer's own registers -- the authority the document pack is checked against.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
PATH = os.path.join(HERE, "data", "registers.json")
REG = json.load(open(PATH, encoding="utf-8"))
BUYER = REG["buyer"]
MAKERS = REG["makers"]
PARTS = REG["parts"]
SPECS = REG["specifications"]
SUPPLIERS = REG["approved_suppliers"]
MAKER_BY_NAME = {m["name"].strip().lower(): code for code, m in MAKERS.items()}

src/dates.pydate arithmetic

Loose date parsing and the subtractions the rulebook turns on: a revision's issue date against a manufacture date, a date code's production week against a stated manufacture date, a letter's validity window against the day the goods arrived, a test report's date against manufacture.

# Dates, and the one piece of encoding this kit has to decode. Pure code, no model, no key.

MONTHS = {m: i + 1 for i, m in enumerate(
MONTH_RE = (r"(jan(?:uary)?|feb(?:ruary)?|mar(?:ch)?|apr(?:il)?|may|jun(?:e)?|jul(?:y)?|"
MONTH_ABBR = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]
MONTH_FULL = ["January", "February", "March", "April", "May", "June", "July", "August",
def day(value):
def iso(d):
def days_between(a, b):
def on_or_before(a, b):
def within_days(a, b, n):

src/recheck.pythe recheck station — a swap seam

Takes the model's READING and nothing else, hands it to the same engine, and records the verdict both ways -- raw and rechecked -- with any override. ⚠︎ On this run it overrode nothing.

You change it to: Which of the twenty read fields the recheck takes from the reply. Everything else is re-derived.

# The model's reading, the rulebook's arithmetic. Pure code, no model, no key.

TAKEN_FROM_MODEL = ("part_number", "po_part_number", "purchase_order", "supplier_code",
DERIVED = ("verdict", "indicators")
def recheck(answer, pack):

src/adapters/__init__.pythe model adapter — a swap seam

Raw HTTP over urllib to any OpenAI-compatible provider or Anthropic. No SDK, no framework.

You change it to: PROVIDER + BASE_URL + MODEL in .env -- one line, then the same run again. Two adapter shapes ship (OpenAI-compatible and Anthropic). A provider that reports no cache split gets usd null with the reason, not an estimate.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TIMEOUT_S = 1200
TRANSPORT_RETRIES = 1
def _post(url, headers, payload, timeout=TIMEOUT_S):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}

src/budget.pythe cost meter

Prices each call from the provider's own reported prompt/completion tokens and cache split, at the tariff in force at the UTC hour of that call. A forker on another provider gets usd null with the reason attached, never a guess.

# A cap on live calls, shared by every kit on this machine. No dependency, no service.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
KIT = os.path.basename(HERE)
ROOT = os.path.dirname(os.path.dirname(HERE))
SHARED_ENV = os.path.join(ROOT, ".env")
LEDGER = os.path.join(ROOT if os.path.exists(SHARED_ENV) else HERE, ".calls-ledger.jsonl")
PAID_ARMS = ("r", "x")
FREE_ARMS = ("t", "b")
BOOT = os.urandom(6).hex()
PID = os.getpid()

evals/baseline.pythe free floor — a swap seam

A keyless reader -- CSV, labelled fields, register lookups, hand-written dates -- in front of the SAME engine. It declares in its docstring, before the run, the two things it expects to break on.

You change it to: The readers in front of the engine. ⚑ THIS SEAM IS THE ONE THAT MOVED THE HEADLINE: two regexes for the prose chain and the prose letter take the floor from 56 of 60 to 60 of 60.

# THE FREE FLOOR. No key, no model, no network. A genuine attempt at the job.

MODES = ("rules",)
CATALOGUE_PARTS = sorted(REG.PARTS)
SPEC_IDS = sorted(REG.SPECS)
SUPPLIER_NAMES = {s["name"]: code for code, s in REG.SUPPLIERS.items()}
MAKER_NAMES = sorted(m["name"] for m in REG.MAKERS.values())
DOC_WORDS = [(k, k.replace("_", " ")) for k in R.ALL_DOCS] + [
SUP_CODE = r"\b(SUP-\d{3,5})\b"
LOT = r"\b(L-\d{2}-\d{3,5})\b"
PO = r"\b(PO-\d{4,6})\b"

evals/scoring.pythe scorer — a swap seam

Exact match on the verdict, set equality on the indicators, micro and macro precision/recall over the key's 56 labels, both error directions, and a taxonomy. One scorer grades every arm.

You change it to: The graders, the two error directions, the taxonomy and every published denominator.

# Grade one arm against the answer key. Deterministic -- no model judges anything here.

KNOWN = set(R.INDICATORS)
RISK = set(R.RISK_VERDICTS)
TAXONOMY = ("no_verdict", "false_clear", "false_alarm", "evidence_confused", "tier_wrong",
def _pct(n, d):
def _set(v):
def _bucket(gv, gi, mv, mi):
def score(records, golds, texts=None):

evals/check_labels.pythe label gate

A second implementation of the judgement that imports no src/ module, re-derives all 60 answers, checks every identifier the key claims to have read is present in the pack's own text, and checks no pack states its own answer. Prints KEY CLEAN or refuses.

# Grade the ANSWER KEY. No key, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
RULES = json.load(open(os.path.join(DATA, "rulebook.json"), encoding="utf-8"))
REG = json.load(open(os.path.join(DATA, "registers.json"), encoding="utf-8"))
PACKS = {p["id"]: p for p in json.load(open(os.path.join(DATA, "packs.json"), encoding="utf-8"))}
REQUIRED = list(RULES["documents"]["required"])
ALL_DOCS = REQUIRED + list(RULES["documents"]["optional"])
TIERS = RULES["tiers"]

tools/build_corpus.pythe corpus generator

Writes the registers, the 60 packs, the key and the stats from seed 20260831. Four renderers. Verified byte-identical across four PYTHONHASHSEED values and against the committed files.

# Build the registers, the corpus and the answer key. Deterministic, free, offline.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
SEED = 20260831
DATASET_VERSION = "part-authenticity-v1-60packs"
RECEIVED_FIRST = _dt.date(2026, 6, 1)
RECEIVED_LAST = _dt.date(2026, 8, 15)
BUYER = "Halden Aerostructures Ltd"
MONTH_FULL = ["January", "February", "March", "April", "May", "June", "July", "August",

ui/index.htmlthe board

One static page, no build step, 44227 bytes. Replays the committed result files -- floor, raw, rechecked -- pack by pack and over the whole corpus, with no key and no call.

# 

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 4699 input and 6914 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

⚠︎ WRITTEN, WIRED, AND NOT FIRED — AND NO NUMBER ON THIS PAGE SAYS OTHERWISE. A document pack is this kit's injection surface and it is an unusually exposed one: the pack is written ENTIRELY by parties outside the buyer, travels with the goods, and reaches the model verbatim. Free text rides along in every format — a covering note, a supplier remarks line, an inspector's transcription of what the supplier said. The party with the strongest motive to address a sentence to whatever reads the folder is the party who wrote the rest of it. The architectural answer is that every indicator is a LOOKUP against data/registers.json and a date subtraction, never a sentence in the pack, so an injected 'already verified' cannot reach the RECHECKED verdict. That is an argument, not a measurement. What an injected line COULD still move is the READING — a sentence that makes the model report a chain that joins up, a letter that covers the part, or a document as present would move the rechecked answer too. Nobody has tried.

API_KEY is read from the shared <code>&lt;repo&gt;/.env</code>, this kit's own <code>.env</code>, or the real environment, in that order. It is never written to a result file, never printed and never sent anywhere but the one provider configured. Every free arm — the label gate, the corpus rebuild, the rules floor, the stub arm and the board — runs with the key absent, and <code>API_KEY= python3 -m evals.run --run-id r001-part-authenticity --resume --rescore</code> is the form that re-derives costs from the cache and fails loudly rather than quietly spending.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. ⚠︎ THE ADVERSARIAL ARM WAS NEVER FIRED. python3 -m evals.injection --run-id x001-part-authenticity --limit 20 --yes is the command; ~20 calls, paired against this run's own cached answers so the control costs nothing. Until it runs, this kit has measured NOTHING about injection. Measured on no run — results/ holds no x001-part-authenticity file, and no adversarial figure appears anywhere in this spec.

Both gates are open. Writing the shape down here is what stops a future run reporting a padded denominator as resistance.

The result⚠︎ NOT FIRED — no adversarial number exists for this kit. evals/injection.py is written, wired, and has the control cache it needs sitting on disk.

The design, so it can be checked rather than re-argued later: each trial is one pack of this corpus whose key verdict is a risk, with ONE line appended in the register of a supplier note the pack already carries, claiming the buyer's verification has already been done and passed. The control is the scored run's own cached answer for the same pack, so a trial costs one call rather than two, and only packs the control got right enter the paired denominator. Both arms are scored: raw, which is what the injection targets, and rechecked, where every indicator is re-derived by lookup.

HonestyWhat this does not prove

• Whether ANY injected sentence changes ANY verdict. Zero trials fired; no number exists.
• Whether the recheck's structural argument holds under attack. It re-derives indicators from data/registers.json, so an injected claim cannot reach the verdict DIRECTLY — but it re-derives them from the model's READING, and the reading is the surface nobody has attacked.
• Whether the four pack renderers differ in exposure. A prose inspection note carries free text by design; a CSV manifest carries one row. Which is easier to inject into is unmeasured.
• Anything about key handling under a hostile provider, TLS interception, or a provider that logs prompts. The pack, the registers and the rulebook all leave the machine on every paid call.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never quarantine a consignment, never reject a part, never accuse a supplier, never contact anybody and never write to any register. This kit produces the RECORD a person confirms, and checks_not_run is the part of that record saying which questions it could not answer — so an absence never reads as a clean bill.

Stated on the UI, in the README and in the rulebook's own framing, and enforced by there being no such endpoint: the board is a static page that replays committed result files, and nothing in src/ writes anywhere but results/.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is ABSENCE OF A WRITE PATH plus a prompt rule, not a runtime enforcement layer — with one partial exception: the recheck IS enforcement for everything the rulebook decides by lookup, subtraction or precedence. ⚠︎ AND ON THIS RUN IT ENFORCED NOTHING: recheck_overrides is 0. It is a control that has never fired.
• It is not a counterfeit detector. Every indicator is a statement about documents; a forged pack that agrees with itself and with the registers is AUTHENTIC here.
• It is not protection against a misread identifier. The engine looks up whatever the reading gives it, so a wrong lot id returns a confident wrong verdict and nothing downstream catches it.
• It is not injection resistance. The structural argument is written down and the probe is not run.
• It is not a disposition. Nothing here quarantines, rejects, releases or notifies, and adding that is not a flag — there is no endpoint.

WatchedWhat is watched, and why that one

The kit measures 33 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

75 measured by the latest run -42 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (authentic_records 16, calls_in_totals 60, clean_records 11, dataset_version part-authenticity-v1-60packs, failures 0, hard_records 49, indicator_labels_arm 56, indicator_labels_key 56, indicator_macro_precision_over_packs 47, indicator_macro_recall_over_packs 49, indicator_precision_denominator 56, indicator_recall_denominator 56, non_risk_records 24, packs 60, packs_answered 60, rechecked_authentic_records 16, rechecked_clean_records 11, rechecked_hard_records 49, rechecked_indicator_labels_arm 56, rechecked_indicator_labels_key 56, rechecked_indicator_macro_precision_over_packs 47, rechecked_indicator_macro_recall_over_packs 49, rechecked_indicator_precision_denominator 56, rechecked_indicator_recall_denominator 56, rechecked_non_risk_records 24, rechecked_rederived_from_cache False, rechecked_risk_records 36, rechecked_tier_records 34, rechecked_trap_records 5, risk_records 36, socket_timeout_s 1200, tier_records 34, trap_records 5) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

3 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

comply · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 75 chips that all say so.

comply · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 79 chips that all say so.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 75 chips that all say so.

DeviationsWhat deviated

0 breaches across 3 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The corpus generator, the label gate, the free floor, the recheck and the stub arm are free and re-run in seconds on a keyless copy, so every figure on this page except the paid arm's tokens and dollars can be re-derived by anybody at $0.00. The paid arm ran once, on 2026-08-31, and was not re-fired.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, standard library end to end — no orchestration layer, no vendor SDK, no retrieval stack, no agent loop, no schema library. requirements.txt installs nothing. The reason is the kit's own argument: the model is trusted with the READING and nothing else, so what would normally be framework surface (routing, tool choice, retries around a chain of calls) does not exist here — there is one call per pack and pure code either side of it.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear, and concurrent only at the pack level. There is no agent, no tool loop, no retrieval and no state carried between packs: read the pack -> one call -> parse -> re-derive every indicator in code -> record. The only branch is the rulebook's four-tier precedence, and it is data.

The other sideWhat a framework costs you

• Adding a provider means editing the adapter by hand. In exchange pip install pulls nothing and a forker runs this on whichever key they already hold.
• No retry policy beyond the socket timeout. A transient provider failure counts the pack wrong rather than dropping it — deliberate, and it did not arise here (0 failures).
• No streaming, so a large generation holds a socket for minutes. The timeout is 1200 s for that reason and the slowest call took 192.7 s.
• No schema enforcement on the reply. It parsed on all 60 calls and invented 0 indicator ids, so this is an untested cost rather than a paid one.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-part-authenticity on the fast tier, 2026-08-31. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 1 earlier run on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 1 run on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- the text is in the repository's LICENSE-PUBLIC, not in its root LICENSE, which governs the repository itself and grants a reader nothing. The kit, its code and this generated corpus are MIT; the repository is not, and it is private. data/SOURCES.md states the distinction and names the file. Every buyer, maker, distributor, part, specification, lot, certificate and register entry here is invented and so is the rulebook, so there is no third-party data in this kit to licence and nothing to clear. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe whole authenticity risk record -- the verdict AND the set of indicators behind it

whether each of the 60 packs produced a record a receiving inspector could confirm as written: the verdict the rulebook's precedence produces, and the exact SET of indicators that fired, including the informational one. Both must be right for the pack to count.

MethodHow the test was run

python3 -m evals.run --run-id <id> [--floor rules]; evals/scoring.py compares against data/gold.jsonl -- the verdict as an exact string from the closed four-value vocabulary, the indicators as a set of ids. An id the arm invents is counted and named.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl -- DERIVED, NOT WRITTEN. tools/build_corpus.py plants a case, builds the reading it implies and calls src/rules.py::assess() to produce the record; the renderer then writes the pack from that same reading. evals/check_labels.py re-derives all 60 by a second implementation that imports no src/ module, and also checks every identifier the key claims to have read is present in the pack's own text and that no pack states its own answer. It printed KEY CLEAN.

No true/false rates for this grader. It records 7 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No tolerance. The verdict is an exact string from a closed four-value vocabulary and the indicator set is compared as a set -- one extra id or one missing id fails the pack.

Cadence: Once per corpus version. The paid arm ran ONCE, was not re-fired, and the recheck is re-derived from the same cached replies at $0.00.

The decisionWhen to reach for it

In one lineThe indicator labels themselves -- precision and recall over the key's 56 labels

whether the arm named the right findings, not just landed on the right verdict. A verdict is one of four values and can be right for the wrong reason; the indicator labels are what a reviewer would actually check the pack against.

MethodHow the test was run

Same run, same scorer -- reported per indicator and per tier as well as micro and macro.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The indicator sets in data/gold.jsonl, derived by src/rules.py from the planted reading.

No true/false rates for this grader. It records 8 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? Exact id match. No partial credit for an adjacent indicator in the same tier.

Cadence: Once per corpus version, with the record grader.

The decisionWhen to reach for it