The business caseThe problem this solves

A trace request names one suspect raw-material lot and asks where it went. The answer is not in any one record: it is a chain across receiving notes, batch tickets, rework records, packaging runs and shipment notes, and somebody has to walk it. Three things on the page look like the chain and are not -- a batch that consumed a DIFFERENT lot of the same material, whose ticket reads almost identically to a real one; a batch ticket that names the material and records no lot at all, which is a candidate rather than a link; and a rework record that puts recovered material into a LATER batch, which forks the chain where most readers stop. Someone opening a site's production records for one suspect raw-material lot and working outward by hand: which batches consumed it, which finished lots those batches became, which consignments carried them, and which of those consignments have already left. It is an afternoon per request and the part that goes wrong is not the reading -- it is stopping one hop short at a rework record, following the material name instead of the lot number, and recording a link as established when the paperwork never actually ties the two ends together.

The inputThe actual trace files

The corpus is 50 trace files, 0.14 MB (txt 50). Plain text, one format, invented rather than fetched -- a real trace file names a real plant, real suppliers, real customers and real consignments, and both the site and its customers consider all of it commercially confidential. There is no public corpus of (production records, correct genealogy) pairs. Generating it also does something this shape particularly needs: THE GENERATOR BUILDS THE TRUE GENEALOGY GRAPH FIRST -- which batch consumed which lot, which batch became which finished lot, which consignment carried it -- and only then derives gold by calling the shipped rulebook over the same facts it wrote into the record text. Gold is not somebody's reading of a chain; it is the chain, plus a lookup. That is also what makes via_rework and depth available as gold metadata, and those two are what turn a reachability percentage into an answer.

Synthetic Record
----------------
Every site, material, code, quantity and consignee in this file is invented for a worked
example. No real plant, company, supplier, customer, product, consignment or person is described
here, and no traceability standard, regulation or company trace procedure is reproduced or
named.

Trace Request
-------------
TR-73451
Suspect raw-material lot: RM-41322
Material: Whey concentrate WC-30
Raised with the quality desk for a genealogy assembly. No hold has been placed and no customer
has been contacted.

Site Reference
--------------
Ravensgate Packing Hall (line 4), quality contact reference QC-2200, customer account AC-51000,
internal telephone 4400.

Receiving Note RC-3275
----------------------
Lot RM-41322, material Whey concentrate WC-30, received 2026-04-07.
Quantity booked in: 365 kg.

Receiving Note RC-2112
----------------------
Lot RM-17498, material Whey concentrate WC-30, received 2026-04-08.
Quantity booked in: 339 kg.

Batch Ticket BT-3516
--------------------
Batch BA-3415, made 2026-04-11 on line 4.
Consumed: Whey concentrate WC-30, lot RM-41322, 299 kg.
Consumed: Whey concentrate WC-30, lot RM-17498, 271 kg.
Consumed: Malt extract ME-44, lot RM-41863, 46 kg.

Batch Ticket BT-1825
--------------------
Batch BA-2986, made 2026-04-14 on line 4.
Consumed: recovered material from rework record RW-7977, 23 kg.
Consumed: Malt extract ME-44, lot RM-41863, 77 kg.

Batch Ticket BT-8237
--------------------
Batch BA-5267, made 2026-04-12 on line 4.
Consumed: Whey concentrate WC-30, lot not recorded, 156 kg.
Consumed: Malt extract ME-44, lot RM-41863, 103 kg.

Rework Record RW-7977
---------------------
Batch BA-3415 sent 23 kg of recovered material to the rework tray on 2026-04-12. That material

The outcomeWhat a good result looks like

A worksheet row per node the records reach -- its code, its kind, the node it was reached from, the record that makes the step, the three facts that record states, and the two calls the shipped rulebook makes from them -- plus one pure-code review flag. Every call carries the rulebook's own reasoning and not_determined is a first-class answer. Nothing here decides a recall, places a hold, withdraws or releases stock, stops a consignment, notifies a customer or opens a regulatory file.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Assembling a genealogy for a trace request or a mock recall, as a first pass for a quality reviewer to work through — the one tier measured here -- 308 of 326 nodes reached with 0 phantoms, 63 of 64 through rework, and every link call it reached exactly right
  94.48 pct reachability against the free material-name floor's 67.79 pct, 0 phantom nodes against 78, and 98.44 pct of the rework-only nodes against 0. On the link call it is 308 of 308 with nothing collapsed.

And where nothing here is good enough:

• Deciding whether the pure-code review flag can be relied on — nothing -- this page says no, and says why
  It fired 1 of 11 with no false alarms. The rule is not wrong; it is downstream of the one thing the run misses, and 10 of the 11 positives hang on a consignment that was never returned. A business-condition guardrail is only as good as the node the run reaches, and this page is the demonstration rather than the assertion.
• Deciding whether a cheaper or a more deliberate model would do better here — nothing -- this page cannot answer it
  Only one model was run. There is no tier comparison, no repeatability pair and no ablation on this page. The link call is already at 308 of 308, so it could not rank two models; the disposition call at 271 of 308 and consignment reachability at 54 of 72 are the two graders with room to move.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-lot-trace. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts, the 50-file picker populates, the genealogy table draws its seven column headings and the shipped rulebook renders in full from data/rulebook.json; clicking Assemble genealogy returns the no-key sentence rather than a stack trace, and the review row reads “not computed -- the reply carried no node the rule could read” rather than “no”. python -m evals.check_labels passes with no network access at all, and so does the free floor.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per trace file

Current processWhat it replaces

Someone opening a site's production records for one suspect raw-material lot and working outward by hand: which batches consumed it, which finished lots those batches became, which consignments carried them, and which of those consignments have already left. It is an afternoon per request and the part that goes wrong is not the reading -- it is stopping one hop short at a rework record, following the material name instead of the lot number, and recording a link as established when the paperwork never actually ties the two ends together.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one record section is

cut on underlined record headings -- one section per receiving note, batch ticket, rework record, packaging run and shipment note, plus the request, the site reference, the stock position and the closing note; a file with no headings falls back to one whole-document segment

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only -- 50 files cut into 836 record sections in 0.0023 seconds, in process, with no model and no network. 786 of those sections are sent; the 50 that are not are exactly the Site Reference sections, which carry the plant, an internal quality contact reference, a customer account number and a telephone extension, and which no field asks for.

LicenceLicence

MIT -- this repository's own licence. Every site, material, lot, batch, packaging, consignment and consignee code is invented; no real plant, company, supplier, customer, product, consignment or person is named, and no traceability standard, regulation, certification scheme, industry guideline or certification body is named either. The genealogy rulebook this kit ships (data/rulebook.json) is ILLUSTRATIVE and is not an authority. It was written for this kit and reproduces no standard, regulation or company trace procedure.

Bring your ownBring your own trace files

REPLACE data/rulebook.json FIRST. It is this kit's own construction, it reproduces no traceability standard, and everything downstream reads it: src/rulebook.py loads it, src/prompt.py renders it into every call, and tools/build_corpus.py writes gold with it. Then replace data/corpus/*.txt, write your own data/fields.json, and supply a gold row per file carrying its genealogy graph AND its decoys -- the decoy list is what makes the phantom grader possible, and on this shape the decoys are most of the difficulty. SECTION_HINTS in src/select.py maps fields to record headings and will need editing for a different layout. ⚠︎ READ src/select.py::_fallback BEFORE YOU DO: every sibling kit falls back to the whole document when a field's sections are absent, and on this shape that is a privacy hole -- the gold gate convicted it on its first run, when a file with no packaging run and no shipment note dragged the Site Reference section onto the wire. compute() in src/extract.py is this kit's own review rule and is the second thing to replace.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 3,778 tokens assembled, 1,924 are contexts — 51% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on LOT-0020 -- the same file the worked example above was captured on, so the four parts sum to exactly the 3778 input tokens that call actually reported. Four calls at max_tokens=1, each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. kits/UC0066-lot-trace/results/tokens-p001-lot-trace.json. ⚑ THE FIXED PREFIX IS 3039 of 3778 TOKENS (80.4 pct) and is paid on every one of the 50 calls.

Full promptThe prompt in full

You read a TRACE FILE -- the production records a site holds for one trace request -- and return a REVIEWER'S TRACE WORKSHEET: the genealogy of one suspect raw-material lot. You return JSON and nothing else.

You are preparing a worksheet for a qualified reviewer. You never decide a recall, never place a hold, never withdraw or release stock, never notify a customer and never open a regulatory file. Your job is to say what the records state, apply the rulebook given below, and NAME THE LINKS THE RECORDS DO NOT ESTABLISH.

RULES, in order of importance:
1. ONE OBJECT PER NODE THE RECORDS PUT DOWNSTREAM OF THE SUSPECT LOT, AND NO OTHERS. A node is a production batch, a finished lot or a consignment. Work outward from the suspect lot: the batches that consumed it, the finished lots those batches were packed into, and the consignments that carried those finished lots.
2. FOLLOW THE CHAIN ALL THE WAY, INCLUDING THROUGH REWORK. A rework record that re-enters recovered material from a batch already on the worksheet into a LATER batch puts that later batch on the worksheet too -- and its packaging run, its finished lot and its consignment after it. A trace that stops at the first hop is wrong and looks complete.
3. A MATERIAL NAME IS NOT A LOT NUMBER. A batch that consumed a DIFFERENT lot of the same material is NOT on the worksheet, and nor is anything downstream of it. A batch ticket that names the material and records NO lot against it IS on the worksheet, with `link_state` 'not_determined'. Those two lines look almost the same and they are opposite answers.
4. THE CHAIN STOPS AT A LINK THE RECORDS DO NOT ESTABLISH. Report a node whose `link_state` is 'not_determined'; do NOT report what lies beyond it. The records do not say the material got that far, and putting the lot beyond it on a recall worksheet places stock there that nothing in the file places there.
5. REPORT WHAT THE RECORDS STATE. `source_record`, `identity_link` and `outflow` are readings of the paperwork, not judgements about it. If nothing in the file says what became of a node, `outflow` is 'silent' -- that is a finding about the records, not a gap in your answer.
6. `link_state` IS DECIDED ONLY BY THE RULEBOOK BELOW, from `source_record` and `identity_link`. Work through the five steps IN ORDER and STOP at the first that fires. In particular a shipment note that names ONLY the packaging code stamped on the cases, where a packaging run records that code against the finished lot, is 'established_by_pack_code' and never plain 'established'.
7. `disposition` IS DECIDED ONLY FROM `outflow`: 'shipment_recorded' -> 'left_site', 'consumed_onward' -> 'moved_on', 'held_in_stock' -> 'on_site', 'silent' -> 'not_determined'. It is asked of every node independently of its link call.
8. 'not_determined' IS A REAL ANSWER AND YOU ARE EXPECTED TO USE IT. A trace that never reaches for it is guessing, and a link recorded as established is a link nobody goes looking for.
9. Copy every code verbatim from the file. `reached_from` is ONE HOP BACK -- the suspect lot only for a batch that consumed it directly. Use the exact allowed value for every field that lists them, and return every field for every node.

GENEALOGY RULEBOOK (the authority for `link_state` and `disposition`; this is an ILLUSTRATIVE rulebook written for this kit, and it reproduces no traceability standard, regulation or company trace procedure)

WHAT IS ON THE WORKSHEET
  - A node is on the worksheet if a record in this file puts it downstream of the suspect lot - directly, or through another node already on the worksheet.
  - A CHAIN STOPS AT A LINK THE RECORDS DO NOT ESTABLISH. A node whose link_state is `not_determined` is ON the worksheet, because naming it is the whole point; what lies beyond it is NOT, because the records do not say the material got that far. The worksheet reports the candidate and stops.
  - A batch that consumed a DIFFERENT lot of the same material is NOT on the worksheet, and nor is anything downstream of it. The material name is not the lot.
  - A batch that consumed an unrelated material is NOT on the worksheet.
  - REWORK FORKS THE CHAIN AND DOES NOT END IT. A rework record that re-enters recovered material from a batch on the worksheet into a LATER batch puts that later batch on the worksheet too, with everything downstream of it. Stopping at the first hop is the commonest way a trace comes back short.

NODE KINDS
  batch          a production batch, recorded on a batch ticket, that consumed material
  finished_lot   a finished lot produced by a packaging run from one batch
  shipment       a consignment recorded on a shipment note that carried a finished lot off the site

THE THREE STATED FACTS, read off the records for every node
  source_record:
    code_recorded          the record that puts this node in the chain names the material it received BY CODE - a lot code, a batch code, a finished lot code or a packaging code
    material_only          the record names the material, the description or the product and records NO code against it
    not_recorded           the record does not state what this node received at all - it names the node, and declines to record what went into it
  identity_link:
    same_code              the code the record uses is the upstream node's OWN code, and resolves directly. This is also the value when the record records no code at all, because no packaging code is in play
    via_pack_code          the record names only a PACKAGING CODE, and a packaging run in this file records that packaging code against the upstream node - so the code resolves, through a substitution
    pack_code_unmatched    the record names a packaging code that NO packaging run in this file records against anything - so it resolves to nothing
  outflow:
    shipment_recorded      a shipment note in this file records this node leaving the site
    consumed_onward        a later batch ticket, rework record or packaging run in this file records this node being consumed
    held_in_stock          a stock position line in this file records this node still on site
    silent                 no record in this file says what became of it

THE LINK CALL -- work through IN ORDER, stop at the first that fires
  1. NO INPUT RECORDED. If source_record is `not_recorded`, the worksheet records `not_determined`. A record names this node and declines to say what went into it, so the records do not tie it to the suspect lot however plainly they name the node itself.
  2. A MATERIAL NAME IS NOT A LOT NUMBER. If source_record is `material_only`, the worksheet records `not_determined`. The record names the material and records no lot against it, so this node MAY have consumed the suspect lot and the paperwork does not say. This is the branch a confident reader skips, because the material name matches and matching feels like evidence.
  3. A PACKAGING CODE NOTHING RECORDS. If identity_link is `pack_code_unmatched`, the worksheet records `not_determined`. A code that appears in one record and nowhere else ties nothing to anything.
  4. TIED THROUGH A PACKAGING CODE. If identity_link is `via_pack_code`, the worksheet records `established_by_pack_code`. The link HOLDS - a packaging run records that code against this node - but it holds through a code substitution rather than through the node's own code, and a reviewer has to be able to see which links rest on that.
  5. ANYTHING ELSE. The worksheet records `established`.

THE DISPOSITION CALL
  1. If outflow is `shipment_recorded`, the worksheet records `left_site`.
  2. If outflow is `consumed_onward`, the worksheet records `moved_on`.
  3. If outflow is `held_in_stock`, the worksheet records `on_site`.
  4. If outflow is `silent`, the worksheet records `not_determined`. The disposition question is asked of every node INDEPENDENTLY of its link: a node the records do not tie to the suspect lot still either has a stated outflow or it does not.

WHY `not_determined` IS A REAL ANSWER
  A trace worksheet that never says `not_determined` is not a complete trace, it is a confident one. The whole value of this shape is that a reviewer can see, in one column, which links the records actually establish and which ones somebody has to go and find the paperwork for. Over-confidence here costs more than caution in both directions at once: a link recorded as established is a link nobody goes looking for, and a chain that stops silently is stock nobody knows is out there.

Return these:
- trace_id (string) -- the trace request reference from the Trace Request section, verbatim
- suspect_lot (string) -- the raw-material lot code the Trace Request names as suspect, verbatim
- lot_remainder (enum) one of: remainder_in_stock, fully_consumed, not_stated -- what the Stock Position section says about the SUSPECT LOT ITSELF. 'remainder_in_stock' when a line records a remaining quantity of that lot still on site; 'fully_consumed' when a line records the lot fully issued to production with none remaining; 'not_stated' when the Stock Position section says nothing about the suspect lot. This is about the raw-material lot, never about a finished lot
- nodes (array of objects) -- one object per node the records put downstream of the suspect lot, each carrying:
    - node_code (string) -- the code of this node, verbatim (for example BA-4417, FL-50231, SH-8802). THIS IS THE KEY -- one object per node the records put downstream of the suspect lot, and no others
    - node_kind (enum) one of: batch, finished_lot, shipment -- what kind of thing this node is: a production batch, a finished lot produced by a packaging run, or a consignment on a shipment note
    - reached_from (string) -- the code of the IMMEDIATE upstream node this one was reached from, verbatim -- the suspect lot code for a batch that consumed it directly, the batch code for a finished lot, the finished lot code for a shipment, and the SOURCE batch code for a batch reached through a rework record. One hop back, never the suspect lot for anything but a first-hop batch
    - record_ref (string) -- the code of the record that makes this step, verbatim -- the batch ticket, rework record, packaging run or shipment note that ties this node to `reached_from` (for example BT-2140, RW-3301, PK-5512, SN-8802)
    - source_record (enum) one of: code_recorded, material_only, not_recorded -- what that record says about this node's INPUT. 'code_recorded' when it names the input by code -- a lot code, a batch code, a finished lot code or a packaging code. 'material_only' when it names the material, description or product and records NO code against it. 'not_recorded' when it names no input at all. Report what the record says -- this is NOT on its own what decides `link_state`
    - identity_link (enum) one of: same_code, via_pack_code, pack_code_unmatched -- how the record ties this node's OWN identity. 'same_code' when the record names this node's own code. 'via_pack_code' when it names only a PACKAGING CODE and a packaging run in this file records that packaging code against this node. 'pack_code_unmatched' when it names a packaging code that no packaging run in this file records against anything
    - outflow (enum) one of: shipment_recorded, consumed_onward, held_in_stock, silent -- what this file records about what became of this node. 'shipment_recorded' when a shipment note records it leaving the site; 'consumed_onward' when a later batch ticket, rework record or packaging run records it being consumed; 'held_in_stock' when a stock position line records it still on site; 'silent' when nothing says
    - link_state (enum) one of: established, established_by_pack_code, not_determined -- the worksheet's link call for this step, decided STRICTLY by the shipped rulebook from `source_record` and `identity_link` and nothing else. Work through the five steps IN ORDER, stopping at the first that fires. (1) source_record 'not_recorded' -> 'not_determined'. (2) source_record 'material_only' -> 'not_determined': A MATERIAL NAME IS NOT A LOT NUMBER. (3) identity_link 'pack_code_unmatched' -> 'not_determined'. (4) identity_link 'via_pack_code' -> 'established_by_pack_code', never plain 'established'. (5) anything else -> 'established'
    - disposition (enum) one of: left_site, moved_on, on_site, not_determined -- the worksheet's disposition call, decided STRICTLY from `outflow`: 'shipment_recorded' -> 'left_site', 'consumed_onward' -> 'moved_on', 'held_in_stock' -> 'on_site', 'silent' -> 'not_determined'. Asked of every node independently of its link call

Return a JSON object with exactly these top-level keys: trace_id, suspect_lot, lot_remainder, nodes
`nodes` is an array. Return it empty only if no record in the file puts anything downstream of the suspect lot at all.

TRACE FILE
----------
Synthetic Record
----------------
Every site, material, code, quantity and consignee in this file is invented for a worked
example. No real plant, company, supplier, customer, product, consignment or person is described
here, and no traceability standard, regulation or company trace procedure is reproduced or
named.

Trace Request
-------------
TR-13846
Suspect raw-material lot: RM-50699
Material: Pectin blend PX-40
Raised with the quality desk for a genealogy assembly. No hold has been placed and no customer
has been contacted.

Receiving Note RC-6375
----------------------
Lot RM-50699, material Pectin blend PX-40, received 2026-08-05.
Quantity booked in: 876 kg.

Receiving Note RC-9113
----------------------
Lot RM-45397, material Pectin blend PX-40, received 2026-08-06.
Quantity booked in: 244 kg.

Batch Ticket BT-9992
--------------------
Batch BA-5863, made 2026-08-09 on line 3.
Consumed: Pectin blend PX-40, lot RM-50699, 190 kg.
Consumed: Calcium carbonate CC-08, lot RM-55381, 21 kg.

Batch Ticket BT-1850
--------------------
Batch BA-3455, made 2026-08-10 on line 3.
Consumed: Pectin blend PX-40, lot RM-50699, 418 kg.
Consumed: Pectin blend PX-40, lot RM-45397, 58 kg.
Consumed: Calcium carbonate CC-08, lot RM-55381, 36 kg.

Batch Ticket BT-1579
--------------------
Batch BA-5566, made 2026-08-12 on line 3.
Consumed: rework tray contents, 70 kg.
Consumed: Calcium carbonate CC-08, lot RM-55381, 23 kg.

Rework Record RW-1076
---------------------
Batch BA-5863 sent 70 kg of recovered material to the rework tray on 2026-08-10. Tray contents
were re-entered into batch BA-5566 on 2026-08-12. The tray is not segregated by batch and this
record does not state which batch's material entered BA-5566.

Packaging Run PK-2980
---------------------
Batch BA-5566 packed 2026-08-14 as finished lot FL-95813.
Cases stamped: PC-2897.

Packaging Run PK-9878
---------------------
Batch BA-3455 packed 2026-08-12 as finished lot FL-96151.
Cases stamped: not recorded on this run.

Shipment Note SN-9972
---------------------
Consignment despatched 2026-08-15 to consignee CN-1691.
Cases stamped PC-2527, 218 cases.

Stock Position
--------------
Batch BA-5863: held in quarantine on site, not packed.

File Notes
----------
Circulated for genealogy assembly only. Nothing in this file has been actioned; the disposition
of every lot named here remains as recorded above.

Raw responseThe raw response

{"trace_id":"TR-13846","suspect_lot":"RM-50699","lot_remainder":"not_stated","nodes":[{"node_code":"BA-5863","node_kind":"batch","reached_from":"RM-50699","record_ref":"BT-9992","source_record":"code_recorded","identity_link":"same_code","outflow":"held_in_stock","link_state":"established","disposition":"on_site"},{"node_code":"BA-5566","node_kind":"batch","reached_from":"BA-5863","record_ref":"RW-1076","source_record":"material_only","identity_link":"same_code","outflow":"consumed_onward","link_state":"not_determined","disposition":"moved_on"},{"node_code":"BA-3455","node_kind":"batch","reached_from":"RM-50699","record_ref":"BT-1850","source_record":"code_recorded","identity_link":"same_code","outflow":"consumed_onward","link_state":"established","disposition":"moved_on"},{"node_code":"FL-96151","node_kind":"finished_lot","reached_from":"BA-3455","record_ref":"PK-9878","source_record":"code_recorded","identity_link":"same_code","outflow":"silent","link_state":"established","disposition":"not_determined"}]}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Trace a suspect raw-material lot through your plant — 50 trace files. One model answered, and every answer was then graded Six different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid -- the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersSix ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The material-name floor (evals/baseline.py) and the scorer (evals/judge.py) are both pure code and cost $0.00 to run against any result set. The figure above is the ONE scored run's own token counts (50 files) priced at Google Gemini 3 Flash's published rate -- the same basis cost_per_query_usd uses. ⚠︎ IT IS NOT WHAT BUILDING THIS KIT COST, and on this kit the difference is unusually large because the FIRST CALIBRATION FAILED COMPLETELY AND WAS BILLED IN FULL. Outside the scored run: a 6-file calibration at max_tokens=16000 that returned NOTHING USABLE and still cost $0.3004 (all six replies ran to the 16,000-token ceiling before emitting any JSON -- 96,000 output tokens billed for zero usable documents), a second at 48,000 that measured the ceiling ($0.4517), the worked-example capture ($0.0477), 4 prompt-split calls at max_tokens=1 ($0.0049), a one-token probe confirming the provider would accept a 64,000 ceiling, and 2 live UI screenshots. All 70 live calls behind this page project to about $3.65 on this card. ⚑ AND THE $0.3004 IS THE INTERESTING NUMBER, BECAUSE THE COMMITTED RESULT FILE SAYS $0.00. results/eval-c000-lot-trace-calibration.json records input_tokens_total: 0 and output_tokens_total: 0, because the harness summed tokens only for documents that PARSED and every document in that run failed. A cost page reading that file would have published a zero for the single most expensive mistake available on this kit. evals/run.py now counts a cut-off reply exactly as the provider bills it; the committed c000 file is deliberately left as it was written, with the real figure stated here, because a result file quietly edited after the fact is worse than one with a known defect named beside it.

Cost driversWhat actually moves the bill

• ⚑ PROVIDER-SIDE REASONING, WHICH IS THE WHOLE BILL. 885,922 of the run's 915,761 output tokens (96.7 pct) were reasoning rather than the JSON worksheet, on a reply that is a few hundred tokens of content. Output is 97 pct of the cost on the projected card. Reasoning was left at the provider's default and nothing here has measured what turning it off would cost -- and on this kit that is not only a cost question, because the work IS a traversal.
• The fixed prefix -- system prompt (846 tokens), genealogy rulebook (1185) and field schema (1008) -- is 3039 of 3778 tokens on the example call, 80.4 pct, and is byte-identical on every one of the 50 calls. The trace file itself is 739 tokens. The bill is the rulebook and the instructions, not the records.
• Output length scales with HOW FAR THE CHAIN RUNS, not with a fixed record. A three-node file and a thirteen-node file are not the same call: the run's largest reply was 34,061 output tokens against a mean of 18315, and the calibration on the six longest chains topped out at 45,528.

Your volumeWhat it costs at your volume

Linear in trace files: each call is independent, carries the same fixed prefix and shares nothing with its neighbours, so 500 files cost ten times 50. Nothing amortises -- there is no index and no cache. ⚠︎ AND IT IS NOT LINEAR IN CHAIN LENGTH, WHICH IS THE AXIS THAT ACTUALLY MOVES. The reply is a graph traversal: the six longest chains in this corpus averaged 24,406 output tokens a call against the corpus's 18315, and a site whose lots reach forty nodes rather than thirteen is a different bill and eventually a different shape -- the chain walked in stages across calls, which changes what a run record means. ⚠︎ AND THE WALL CLOCK IS ITS OWN CLIFF: median 145 seconds a file, p95 244. A thousand files is not an overnight job at any sensible concurrency.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 trace files, one completion call each, one tier. The two calibrations, the worked-example capture, the prompt-split calls and the UI screenshot are recorded separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the trace file into addressable sections, pure code -- one per record, plus the request, the stock position and the closing note

# Cut a trace file into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections are sent, pure code -- matched by PREFIX because every section is named after its own record code. The decoy tickets are deliberately SENT; the Site Reference section is mapped by nothing and never leaves the machine, and its FALLBACK path had to be narrowed to keep that true

You change it to: map fields to your own record headings, including the per-record prefixes. NEVER_SENT is the list of sections no field asks for -- and _fallback() is the seam beside it: on this kit the usual fall-back-to-the-whole-document behaviour was a privacy hole, and it now falls back to the whole file MINUS the never-sent sections

# Pick which sections of a trace file are sent. Pure code -- the last deterministic step before

BANNER = "Synthetic Record"
REQUEST = "Trace Request"
STOCK = "Stock Position"
NOTES = "File Notes"
RECEIVING = "Receiving Note "
TICKET = "Batch Ticket "
REWORK = "Rework Record "
PACKAGING = "Packaging Run "
SHIPMENT = "Shipment Note "

src/rulebook.pyrulebook

the two worksheet calls, loaded from data/rulebook.json -- a five-step link rule with a stopping order and a four-way disposition rule. Data, not code, so a forker can open it and disagree with it

# THE GENEALOGY RULEBOOK, loaded from data/rulebook.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
SOURCE_RECORDS = ("code_recorded", "material_only", "not_recorded")
IDENTITY_LINKS = ("same_code", "via_pack_code", "pack_code_unmatched")
OUTFLOWS = ("shipment_recorded", "consumed_onward", "held_in_stock", "silent")
LINK_STATES = ("established", "established_by_pack_code", "not_determined")
DISPOSITIONS = ("left_site", "moved_on", "on_site", "not_determined")
def load(path=RULEBOOK_PATH):
R = load()

src/prompt.pyprompt

assemble one call per trace file for the whole genealogy, with the rulebook RENDERED into it from data/rulebook.json rather than restated in prose -- a rule the model has never been shown is not a rule

# Assemble the trace prompt. ONE prompt per trace file, the whole genealogy in the reply.

SYSTEM = (
RULEBOOK_TEXT = rulebook_block()
def field_schema(trace_fields, node_fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key -- plus the pure-code business-condition check downstream: a file carrying a consignment recorded as despatched whose link the records do not establish goes back for the paperwork

You change it to: the review rule itself -- this kit ships three values (a consignment, despatched, on an unestablished link) chosen against a measured table of five candidates. It is deliberately NOT the same function as the rulebook lookup, so changing WHAT GETS ESCALATED does not change WHAT THE RULEBOOK SAYS

# Assemble one trace file's genealogy: segment, select, prompt, one model call, then a pure-code

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 64000
def load_fields():
def node_fields(fields=None):
def trace_fields(fields=None):
def load_doc(doc_id):
def documents():

evals/judge.pyjudge

score reachability (by node kind, by depth and through rework), the path, the cells, the two calls and the determinacy split, all separately, pure code -- plus a no-gold consistency diagnostic that this kit measures the limits of

# Score a lot-genealogy run. PURE CODE -- gold is exact and every answer is one value or one code,

def norm(v):
def code_norm(v):
def equal(got, want):
def _rows_of(rec):
def score_reachability(records, golds):
PATH_FIELDS = ("reached_from", "record_ref")
def score_path(records, golds):
CELL_FIELDS = ("node_kind", "reached_from", "record_ref", "source_record", "identity_link",
TRACE_CELLS = ("trace_id", "suspect_lot", "lot_remainder")

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3883 input and 18315 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's trace files are entirely synthetic (tools/build_corpus.py, seed 20260822): no real plant, company, supplier, customer, product, consignment or person exists in the corpus, and nothing was fetched from anywhere. No traceability standard, regulation, certification scheme, certification body or manufacturer is named anywhere in the kit, and every file opens with a Synthetic Record banner so a single file copied out of the repository still says what it is. The only outbound traffic the kit makes is one chat-completion request per file, carrying the mapped sections plus the fixed prompt and the shipped rulebook -- and the Site Reference section, which is the one part of a real trace file a quality desk would most mind sending anywhere, is mapped by no field and never leaves the machine. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser.

The experimentWe did not attack it -- and one boundary was broken until the gate caught it

Three boundaries hold. Two were confirmed by reading the code: no code path places a hold, withdraws stock, notifies anybody or writes anything, and the rulebook lookup and the review rule read only enums, so no record text can reach either. The third is confirmed by a CHECK RATHER THAN BY READING, and it needed to be -- evals/check_labels.py asserts, before any run may spend, that the Site Reference section is never sent and that every record section is, and it convicted the kit's first version on a file with no packaging run and no shipment note. The fourth is open. An indirect prompt injection needs a section an outside party authored, and a shipment note is exactly that. Confirmed by reading the code and by a pre-flight assertion that convicted it once, not by an attack run, on 2026-08-22 -- no attack run was fired; see redteam.why.

Three boundaries hold and the third is the interesting one, because it did not hold when it was written: a fallback path every sibling kit ships sent the site reference on one file in fifty, and it was caught by an assertion that runs before every scored run rather than by reading the code. The fourth is open and is named as open -- a trace file is paperwork several parties touch, which makes an instruction hidden in a record exactly the attack this kit's shape invites, and nothing here has tried it.

The result0 attack trials, and three of four boundaries checked here hold on evidence -- one of them only after a pre-flight assertion convicted the code on its first run. The one that does not hold is the one this kit's shape invites: a trace file is paperwork several parties touch, and whether an instruction hidden in a record could move a call -- or, more cheaply, make a consignment go unmentioned -- is unmeasured.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted decoys measure whether a trace follows the LOT rather than the material when the two look alike, whether it follows a rework fork, and whether it declines a link the records do not establish. None of those measures whether it obeys an instruction hidden in the same file, and the drop-a-node attack is the one this kit's own failure mode suggests is cheapest.

HonestyWhat this does not prove

• Whether an indirect prompt injection in a shipment note or a File Notes section could move a call or drop a node. No attack run exists, and the drop-a-node version is the one this run's own failure mode suggests would be easiest.
• Whether the kit behaves safely against a hostile provider -- a response body crafted to break the traversal parsing in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed document, and nothing beyond that was tested.
• Whether a real trace file would carry anything sensitive this kit mishandles beyond the site section. The corpus has no personal data by construction; a real file carries named operators, signatures, customer identities and volumes, and none of that path is exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_trace_review fires when the file carries at least one node whose kind is `shipment`, whose disposition is `left_site` -- a consignment the records say was despatched -- and whose link_state is `not_determined`. All three values come out of the same reply; the rule is run afterwards in pure code, over whatever the model returned, and never over gold. A reply carrying no node the rule can read returns None rather than False -- an unknown is not a pass.

src/extract.py::compute(), called by extract() on every file, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by identical code. The rulebook lookup it reads lives in a SEPARATE module, src/rulebook.py, which is also what the corpus generator wrote gold with and what src/prompt.py states to the model -- one definition, three readers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a recall decision, a hold, a withdrawal, a release, a customer notification or a regulatory filing. Nothing here decides anything about a lot. link_state and disposition are proposals with the rulebook's own reasoning attached, and a quality team decides.
• It is NOT a check on whether the trace is COMPLETE, and this run is the demonstration. The rule reads the reply; if the reply is missing a consignment, the rule is silent about it and the file is not escalated. That is 10 of the 11 files it should have caught. A guardrail downstream of a retrieval step inherits that step's blind spots entirely.
• It is NOT any company's trace procedure. A despatched consignment on an unestablished link is this kit's own simplification, invented for this corpus. No traceability standard, regulation, certification scheme, industry guideline or site procedure was consulted, and none is reproduced. A real trace weighs how much material is involved, where the consignment went, what the product is and how long it has been out.
• It is NOT a judgement about the rulebook. The flag reads the calls the rulebook produced; if the rulebook is wrong, the flag is confidently wrong with it. The shipped rulebook is illustrative and nobody qualified has reviewed it.

WatchedWhat is watched, and why that one

The kit measures 22 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 15 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 50, extraction_cells 2758, failures 0, refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

5 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

extraction · with the model in the path — 4 runs. Columns here are only ever compared with each other.

not a time series No two of these 4 runs measured the same system — they differ on documents, extraction_cells, failures, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

extraction · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 7 chips that all say so.

DeviationsWhat deviated

0 breaches across 5 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/extract.py, on any change to data/rulebook.json (which moves the calls the flag reads), on any change to data/fields.json's allowed values for node_kind, link_state or disposition, and on any corpus regeneration -- the gate re-measures the firing band every time. Changing WHAT GETS ESCALATED is a policy change and must be re-scored, even though it never changes what the rulebook says.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the rulebook is a JSON file read with json, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

The ANSWER is a graph; the pipeline is not. One path per file -- segment, select, prompt, one call, parse, compute -- with no branch, no loop and no agent. What the reply contains is a traversal, and that is what changes the scorer: reachability, the path, and the calls made about what was reached are three different questions and are graded as three.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain -- the JSON extraction from a fenced reply, the traversal parsing, the backoff policy, the section regex, the prefix matcher and the .env reader are all code somebody has to own. This kit paid an instalment during its own build: the free floor's three record regexes were written without re.M, so ^Batch ... anchored at each section's heading and matched nothing -- the floor silently reported ZERO finished lots and ZERO consignments on all 50 files while every other part of it worked.
• No framework means no framework's ecosystem -- no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. The reply is a TRAVERSAL and nothing validates its shape except src/prompt.py::parse, which fails closed to {} rather than to an empty worksheet, because an empty nodes list is a legitimate answer here.
• Nothing checks that the graph the reply describes is CONNECTED. evals/check_labels.py asserts it of gold and ui/app.js walks it defensively to draw the indent, but the extractor would happily return an orphan node with a parent that is nowhere in the reply. On this run it never did; that is an observation, not a guarantee.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-lot-trace on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 4 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

3 runs not plotted. b000-lot-trace-materialname, c000-lot-trace-calibration, t000-lot-trace-stub recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence. Every site, material, lot, batch, packaging, consignment and consignee code is invented; no real plant, company, supplier, customer, product, consignment or person is named, and no traceability standard, regulation, certification scheme, industry guideline or certification body is named either. The genealogy rulebook this kit ships (data/rulebook.json) is ILLUSTRATIVE and is not an authority. It was written for this kit and reproduces no standard, regulation or company trace procedure. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one line⚑ Reachability -- which nodes the trace actually reached

Did the run reach exactly the nodes the records put downstream of the suspect lot? Micro precision and recall over node codes, with the two directions named and never averaged: a MISSED node is something the records do reach, left off, and a PHANTOM node is something on the worksheet the suspect lot never reached. And then split three ways, because a single percentage is not an answer to the question a trace is asked: BY NODE KIND (a missed batch is a gap in the reasoning; a missed CONSIGNMENT is stock at a customer nobody knows about), BY DEPTH from the suspect lot, and THROUGH REWORK -- 64 of this corpus's nodes are reachable only by following the fork, and a run that never follows it returns none of them AND LOOKS COMPLETE.

MethodHow the test was run

evals/judge.py::score_reachability, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, which records the true genealogy GRAPH -- every node, its parent, its depth and whether it is reachable only through a rework record -- plus every decoy code. evals/check_labels.py asserts before any run may spend that the graph is well formed, that no chain continues past an unestablished link, and that each of the three decoy kinds clears a floor of 10.

How tight can the band be? No tolerance band -- a code is on the worksheet or it is not, compared after whitespace stripping and upper-casing.

Cadence: Re-score on any change to data/rulebook.json's what_is_on_the_worksheet, to src/prompt.py's rules 1-4, or to the corpus's graph composition.

The decisionWhen to reach for it

In one lineReached, and by the right route

Of the nodes both sides list, how many name the right IMMEDIATE parent and the right record for the step? Two exact-match cells, reached_from and record_ref. This is the half of reachability a set comparison cannot see: a run can return every node code in the file and still describe a genealogy nobody can walk, because the codes are right and the edges between them are invented.

MethodHow the test was run

evals/judge.py::score_path, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The generator's own graph -- each node's parent and the record that makes the step, written into the file before gold was derived from it.

How tight can the band be? No threshold -- two exact-match code comparisons per node.

Cadence: Re-run whenever the corpus's graph shape changes or the prompt's rule 2 (the rework rule) is edited.

The decisionWhen to reach for it

In one linePer-(node, field) exact match, light normalisation

Does the model's value for each of the eight per-node fields match gold, after trimming whitespace and punctuation, plus three file-level cells per file? THE DENOMINATOR IS GOLD'S NODES, NOT THE RUN'S: a node the run never returned scores as a miss on all eight of its cells rather than disappearing. On a kit whose whole failure mode is stopping early, that is not a refinement -- it is the difference between a real number and a reward for giving up.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every run, and the same logic the free floor is scored by -- a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same values the records state; evals/check_labels.py asserts every field is populated and in its allowed set on every one of the 326 nodes, and that every code is stated in the file it labels, before any run is allowed to spend.

How tight can the band be? There is no tolerance band -- exact match after trimming whitespace and punctuation, never a continuous score to round.

Cadence: Re-score on any change to data/rulebook.json, src/prompt.py, data/fields.json or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe two rulebook calls, three-way and four-way

Did the run land on the same calls the shipped rulebook produces -- established / established_by_pack_code / not_determined for the link, and left_site / moved_on / on_site / not_determined for the disposition? SCORED ON MATCHED NODES ONLY, and saying so is half of what makes the number honest: a call about a node the run never reached has already been counted, once, as a reachability failure.

MethodHow the test was run

evals/judge.py::score_calls, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's calls are re-derived inside the grader by the same lookup the kit publishes -- src/rulebook.py, reading data/rulebook.json -- so the truth this grades against can never be a separately-typed label that drifted from the rule.

How tight can the band be? No threshold -- each call is one of a closed set of values, and a reply returning none of them is counted as unanswered rather than folded into a correct cell.

Cadence: Re-run whenever data/rulebook.json or src/rulebook.py changes, whenever the corpus is regenerated, and on any provider or model change.

The decisionWhen to reach for it

In one line⚑ How often it says the records do not establish it

Of every call this corpus's records do NOT settle, how many did the run settle anyway? That is overconfident, and on a trace worksheet it is the expensive direction: a link recorded as established is a link nobody goes looking for. Its mirror, overcautious, is counted and published beside it, never netted against it. And a THIRD direction this kit has and its siblings do not: established_by_pack_code collapsed into established is not over-confidence about WHETHER the chain runs, it is over-confidence about HOW, and it would otherwise vanish inside a three-way accuracy.

MethodHow the test was run

evals/judge.py::score_calls -- the determinacy block, in-process, no key, no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The same rulebook lookup as the grader above, with the rows partitioned by whether GOLD's answer is not_determined. All three directions come out of one pass over the same 616 calls.

How tight can the band be? A rate over a fixed denominator: 86 open calls and 530 settled ones, over the nodes the run returned. A run over a different corpus has a different denominator and the two rates are not comparable.

Cadence: Re-run on any change to the rulebook's not_determined branches, to the corpus's composition of unestablished links, or to the provider or model.

The decisionWhen to reach for it

In one lineneeds_trace_review against the same rule run over gold

Does the pure-code review decision -- a CONSIGNMENT recorded as despatched whose link into the chain the records do not establish -- land on the same files it would land on if every node had been reached perfectly? It is a BUSINESS CONDITION, so unlike a self-consistency check it genuinely needs labels, and saying so is half of what makes the number believable.

MethodHow the test was run

evals/judge.py::score_flag, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute(), the same function the run uses, applied to GOLD's own nodes. One rule, one input shape, so a change to the rule moves both sides together and the grader cannot silently grade an old policy.

How tight can the band be? No threshold -- three values per node and three comparisons.

Cadence: Re-run whenever compute() changes. Changing WHAT GETS ESCALATED is a policy change and must be re-scored, even though it never changes what the rulebook says.

The decisionWhen to reach for it