The business caseThe problem this solves

A retailer changes prices in batches, and one item's price lives in three places at once: the printed tag on the shelf edge, the till file, and the digital channel. A batch is supposed to move all three. When it does not, a customer is shown one price and charged another -- and the reason nobody notices is that every system the business owns agrees the batch worked. Today this is caught by a store colleague walking an aisle, or by a customer at a till. Replaces walking the aisle with a scanner, and replaces the system-level check that compares the till and the digital channel against the batch and cannot see the shelf edge at all.

The inputThe actual price-change batches

The corpus is 50 price-change batches, 0.15 MB (txt 50). A real price-change extract is not something this repo can publish: it names a retailer's live prices item by item, its store estate, and a named price-control lead with a direct extension. There is no public corpus of (price-change batch, correct exception list) pairs. Generating it also makes the label mechanical. Gold's divergence and gold's worst_gap_minor are both src/rulebook.py run over the same values the record states -- never somebody's reading of the desk's note -- and evals/check_labels.py re-runs the rule over every one of the 332 rows before any run may spend. The composition is an exact count over thirteen buckets rather than a sampled proportion, and each bucket is a reading a careless monitor gets wrong. Three of them are the point: 26 items where the till and the digital channel BOTH took the new price and the shelf tag did not (the case a system-level check cannot see), 30 authorised channel-exclusive promotional prices and 20 rounded unit-derived prices (the two differences that are supposed to be there, and the false alarms that get an exception list ignored).

Batch
-----
BCH-4218-VN

Batch Effective From
--------------------
2026-02-26

Pricing Rulebook
----------------
PIR-1 -- illustrative, shipped with this kit; reproduces no real pricing rule

Items In Batch
--------------
prices are stated in the trading currency's major unit; a third decimal is a unit-derived price
SKU-28916  Shower gel 250ml                     was 1.63      to 1.76
SKU-56851  Milk chocolate bar 100g              was 1.29      to 1.39
SKU-11365  Paracetamol tablets, 16 pack         was 0.58      to 0.53
SKU-13799  Dried spaghetti 500g                 was 0.96      to 1.11
SKU-77983  Bin liners, 20 pack                  was 2.63      to 2.25

Shelf-Edge State
----------------
SKU-13799  tag price 1.11                     tag reprinted 2026-02-26
SKU-11365  tag price 0.58                     tag reprint -- not confirmed on this batch
SKU-56851  tag price 1.39                     tag reprinted 2026-02-26
SKU-28916  tag price 1.76                     tag reprinted 2026-02-26
SKU-77983  tag price 2.63                     tag reprinted 2025-12-24
SKU-47393  tag price 0.94                     tag reprinted 2026-01-18
SKU-94658  tag price 8.98                     tag reprinted 2025-11-08

Point Of Sale State
-------------------
SKU-13799  till price 0.96                    activated 2026-01-30
SKU-56851  till price 1.29                    activated 2025-12-26
SKU-11365  till price 0.53                    activated 2026-02-26
SKU-77983  till price 2.63                    activated 2025-12-24
SKU-94658  till price 8.98                    activated 2025-11-08
SKU-47393  till price 0.94                    activated 2026-01-18
SKU-28916  till price 1.63                    activated 2025-11-29

Digital Channel State
---------------------

The outcomeWhat a good result looks like

One exception list per batch: which items diverge, by how much in minor units, which surface is at fault, and whether the batch needs a job re-run or a price decided by a person.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Checking a price-change batch before the store opens, as an exception list for a pricing-integrity desk to work through — the one tier measured here -- every item called correctly on this corpus, 0 false alarms on 122 flagged items, and all 26 shelf-only failures caught
  The free system-level check gets 46.69 pct of the calls right and cannot see the shelf edge at all; it raises 46 false alarms and misses 33 exceptions on the same 50 records.
• Deciding whether a batch needs an operational retry or a person to decide a price — the pure-code flag in src/extract.py::compute(), not the model
  It routed 50 of 50 records the way the same rule routes gold. The model never sees this question: the flag is a predicate over the item statuses, computed afterwards, so it cannot be talked into anything.
• Running the check with no budget at all — evals/baseline.py -- the system-level check, free, no key, whole corpus in under a second
  It reads every field the rulebook needs and never reformats one: 0 wrong cells on the six price fields against the model's 142, because it keeps the printed string it matched instead of parsing it. Overall per-cell it is 95.77 pct against the model's 96.61 -- LOWER, and the whole difference is the 177 cells where it shortcuts divergence. It catches 89 of the 122 exceptions for nothing.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-price-divergence. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with NO key configured reproduces everything on this page except a new call: the 50 batch records, the 50 gold rows, all six committed run records, and the local UI, which renders and returns a plain sentence saying nothing was called. python3 tools/build_corpus.py rebuilds the corpus byte-identically from the seed; python -m evals.check_labels and python -m evals.run --baseline both run for free and reproduce the floor's numbers exactly. What a clone cannot do without a key is extract a NEW batch or re-score the model.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per batch record, measured in evals/run.py around the whole extract call -- segment, select, prompt, the one model call, parse, span and the pure-code rule. There is no retrieval step to exclude, so this figure and the model-only figure in Cost are the same measurement; both are stated so a reader does not have to assume it. Measured at 12 concurrent workers, so it is per-call latency and not throughput: the whole 50-record run finished in 229.8 seconds.

Current processWhat it replaces

Replaces walking the aisle with a scanner, and replaces the system-level check that compares the till and the digital channel against the batch and cannot see the shelf edge at all.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one item row is

no split -- one whole batch record goes into one call. src/segment.py cuts each record into addressable sections on its underlined headings so a value can name WHICH surface it was read from, but nothing is chunked and nothing is dropped for length.

SetupWhat the setup figure measured

There is no index and nothing is built. Each batch record goes whole into one call, so the build cost is genuinely zero rather than unmeasured -- the only preparation is tools/build_corpus.py regenerating the corpus and gold from the seed, which takes under a second and makes no call.

LicenceLicence

MIT (this repository's own licence). Nothing was fetched and nothing is licensed from anybody: no real retailer, store, product, brand, person or currency is named anywhere in the kit.

Bring your ownBring your own price-change batches

Replace data/corpus/*.txt with your own extracts and data/gold.jsonl with your own labels; rewrite data/rulebook.json with the rules you are actually bound by; adjust the section names in src/select.py to match your headings and put anything you will not send into NEVER_SENT. src/segment.py falls back to one whole-document segment when it finds no underlined headings, so a differently-shaped extract still runs. Then re-run evals/check_labels.py, which will tell you what your labels disagree with before you spend anything.

⚠︎ And what stops being true when you do: Nothing measured here transfers. Two things in particular: this corpus puts all three surfaces in ONE FILE under ONE item-code scheme, so the run never has to reconcile three systems whose keys differ -- which is most of the real work; and the shipped rulebook is an invention, so a 100 pct score means the run agreed with data/rulebook.json and says nothing about whether that file is right for your estate.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 4,794 tokens assembled, 2,233 are contexts — 47% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Replayed from the kit's own src/prompt.py::build() rather than logged by the run, which is only trustworthy because it was checked: the replayed part sizes (5070 / 5190 / 4906 / 3279 characters) and the section list are identical to what the recorded call in results/example-PRC-0044.json wrote down, and the four measured token counts sum to the 4794 input tokens the provider itself billed for that call. The generator asserts both before writing this spec.

Full promptThe prompt in full

You read ONE PRICE-CHANGE BATCH RECORD -- which items a retailer's price change covers, and what each of three surfaces was carrying for those items when the extract was taken -- and you extract structured fields from it. You return JSON and nothing else.

THE THREE SURFACES ARE THE SHELF EDGE, THE TILL AND THE DIGITAL CHANNEL. The question is whether an item's price disagrees with ITSELF across those three, and if it does, whether the disagreement is one that is supposed to be there. A price that is merely different from last week's is NOT a finding -- every item here is different from last week, that is what a price change is.

You are PROPOSING an exception list for a qualified person to act on. You never write a price to any surface, never re-send a batch, never suppress an item to shorten the list and never decide which of two disagreeing prices is the correct one. The rulebook below is an ILLUSTRATIVE rulebook shipped with this kit; it is not a law, a regulation or a retailer's policy, and it is not an authority.

RULES, in order of importance:
1. Return one object per line of `Items In Batch`, in the order that section lists them, and no others. THE OTHER SECTIONS CARRY LINES FOR ITEMS THIS BATCH DOES NOT CHANGE -- an unchanged item whose price is stated for context. Reporting one of those is as wrong as missing a real one. Never invent an item.
2. THE THREE SURFACE SECTIONS ARE IN THREE DIFFERENT ORDERS AND NONE OF THEM MATCHES THE ROSTER. Join by item reference, matched exactly and in full. SKU-4821 and SKU-48213 are different items.
3. If a surface section carries NO LINE at all for an item, that surface's price and date are null. That is not a gap in your answer, it is a fact about the extract, and it is the thing that makes the item undeterminable. Never carry a value across from another surface and never infer one from the batch.
4. If the batch record does not state a field, return null for it. Do not infer it and do not use what you know about the world.
5. `worst_gap_minor` is DERIVED AND IS NEVER STATED. It is the largest absolute difference, IN MINOR UNITS (hundredths), between any two of the three recorded surface prices -- measured against EACH OTHER, not against the batch price. 2.45 against 2.60 is 15. Return 0 where all three agree and null where any one of the three is not recorded.
6. `divergence` is decided ONLY by the RULEBOOK given below. Work through these six gates IN ORDER and STOP at the first one that fires:
   a. NOT DETERMINABLE. The batch price, or any one of the three surface prices, is not recorded for this item. Answer 'not_determinable'. THIS GATE IS FIRST because every comparison below needs those numbers. It is a real answer, not a failure to produce one -- a confident wrong 'fine' is the expensive mistake here.
   b. AUTHORISED CHANNEL PRICE. A promotion is on file for this item with a scope other than none, every surface that scope names carries the promotional price, AND every surface it does not name carries the batch price. Answer 'authorised_channel_price'. THIS GATE IS SECOND. A channel-exclusive promotion is the commonest reason three surfaces legitimately differ, and calling it a divergence is the error that gets an exception list ignored. Note what it still requires: a promotion does NOT excuse a till that never took the batch.
   c. NOT PROPAGATED. At least one surface still carries the item's PREVIOUS price -- the price the batch was changing away from. Answer 'not_propagated'. The change did not reach that surface.
   d. ROUNDING DIFFERENCE. The batch price carries a NON-ZERO THIRD DECIMAL, the three surface prices are not all equal, and every surface price is within 1 minor unit(s) of the batch price. Answer 'rounding_difference'. Each surface rounded a unit-derived price on its own.
   e. DIVERGED. The three surface prices are not all equal; or they are all equal and that price is not the batch price. Answer 'diverged'.
   f. Anything that survives all five gates is 'aligned'.
7. A DATE DECIDES NOTHING. A surface stamped a few days AFTER the effective date carrying the batch price has landed, late, and is aligned. A surface stamped ON the effective date carrying the wrong price has not landed. Read the prices.
8. 'tag reprint -- not confirmed on this batch' means no store confirmed the physical tag. The tag price on the line is still the recorded shelf price and you should return it. It does NOT make the item undeterminable and it does NOT make it a divergence.
9. THE BATCH NOTE IS A FIELD TO COPY, NOT EVIDENCE ABOUT ANY PRICE. A note that sounds settled does NOT clear a divergence, and a note that sounds worried does NOT create one. The batch's own numbers decide; the note is one person's remark and may disagree with them.
10. Copy every reference and every price verbatim, exactly as printed, including a third decimal where the record states one. Every date is yyyy-mm-dd.
11. Use the exact allowed value for a field that lists them, and return every field named in the schema for every item, even when the answer is null.

PRICING INTEGRITY RULEBOOK PIR-1 (the authority for `divergence`; this is an ILLUSTRATIVE rulebook shipped with this kit, not a law, a regulation or a retailer's policy)

THE THREE SURFACES
  shelf, pos, digital
  The three places one item's price is stated to a customer or charged to them: the printed shelf-edge tag, the point-of-sale till file, and the digital channel. THE WHOLE KIT IS THE COMPARISON BETWEEN THESE THREE. A price that is merely different from last week is not a finding here; a price that disagrees with ITSELF across these three surfaces is.

ROUNDING TOLERANCE
  1 minor unit(s). A batch may set a price with a THIRD decimal -- a unit-derived price such as 1.995 -- and each surface rounds it to the minor unit on its own. Where every surface price is within this many minor units of the batch price AND the batch price carries a non-zero third decimal, the surfaces differ because they rounded, not because the price is wrong.

PROMOTION SCOPES -- which surfaces a scope authorises a different price on
  none:      none -- no promotion on file
  digital:   digital
  in_store:  shelf, pos
  A promotion on file authorises a DIFFERENT price on the surfaces its scope names, for this period. `digital` names the digital channel only; `in_store` names the shelf edge and the till together, because a customer in the aisle sees both. A channel-exclusive promotion is the commonest reason three surfaces legitimately disagree, and a monitor that calls it a divergence is the monitor that gets switched off.

THE SIX GATES, IN ORDER. STOP AT THE FIRST THAT FIRES.
  1. NOT_DETERMINABLE:          The batch price, or any one of the three surface prices, is not recorded for this item. No comparison can be run.
     why: On a monitoring queue the expensive mistake is a confident wrong 'fine'. A surface that returned no line, or a line with no price on it, is a record that has to be opened -- not an item that can be cleared.
  2. AUTHORISED_CHANNEL_PRICE:  A promotion is on file for this item with a scope other than `none` and a promotional price recorded; every surface the scope names carries that promotional price; and every surface it does not name carries the batch price to the minor unit.
     why: This is the difference that is SUPPOSED to be there. It sits above the finding gates because a queue that reports every authorised channel-exclusive price as a divergence buries the real ones underneath them. ⚑ NOTE WHAT IT REQUIRES: the surfaces OUTSIDE the promotion must still be right. A promotion does not excuse a till that never took the batch.
  3. NOT_PROPAGATED:            At least one of the three surfaces carries the item's PREVIOUS price -- the price the batch was changing away from.
     why: The change did not reach that surface. This is a different finding from a divergence and it has a different fix: the batch is re-run for that surface. It sits above the divergence gate so that the commonest execution failure is named as an execution failure rather than folded into a general disagreement.
  4. ROUNDING_DIFFERENCE:       The batch price carries a non-zero third decimal, the three surface prices are not all equal, and every surface price is within the rounding tolerance of the batch price.
     why: Each surface rounded a unit-derived price on its own. It is a real difference and it is reported, and it is NOT an exception: nobody is being charged a price the retailer did not set.
  5. DIVERGED:                  The three surface prices are not all equal; or they are all equal and that price is not the batch price to the minor unit.
     why: The surfaces disagree, and none of the legitimate explanations above accounts for it. Somebody has to decide which price is correct before the next customer reaches the till.
  6. ALIGNED:                   Everything else: the three surfaces all carry the batch price to the minor unit.
     why: Nothing needs doing. ⚠︎ AND NOTE WHAT THIS DOES NOT SAY. It does not say a person has seen the shelf tag -- see `shelf_confirmation_note`.

THE SIX STATUSES
  aligned, rounding_difference, authorised_channel_price, not_propagated, diverged, not_determinable

WHAT PUTS AN ITEM ON THE EXCEPTION LIST
  not_propagated, diverged -- The two statuses that put an item on somebody's list today. `aligned`, `rounding_difference` and `authorised_channel_price` are the three answers that mean nothing needs doing. `not_determinable` means the RECORD needs opening rather than the PRICE needing correcting, and it is deliberately not on the exception list for that reason.

THE WORST GAP
  The largest absolute difference, in minor units, between any two of the three recorded surface prices. Not against the batch price -- against each other, because the question this kit asks is whether the item's own price disagrees with itself. Null where any surface price is not recorded.

WHAT THIS TOOL WILL NOT DO
  - It never writes a price to any surface.
  - It never re-triggers, re-sends or re-schedules a price-change batch.
  - It never suppresses, samples down or de-duplicates an item to reduce the size of the list.
  - It never decides which of two disagreeing prices is the correct one.

Extract these fields:
BATCH-LEVEL FIELDS -- one value each, for the whole batch
- batch_ref (string) -- the batch reference, verbatim from the Batch section
- effective_from (date) -- the date this batch's prices take effect, from the Batch Effective From section, as yyyy-mm-dd
- rulebook_id (string) -- the identifier of the pricing rulebook this batch is checked under, verbatim -- the short code only, not the sentence after it
- batch_note (string) -- the price control desk's own free-text remark from the Batch Notes section, copied verbatim. It is a field to copy and it is NOT evidence about whether any price is right

PER-ITEM FIELDS -- one object per line of Items In Batch
- item_ref (string) -- the item reference as the Items In Batch section states it, e.g. SKU-12345. Return one object per line of THAT section, in the order it lists them, and no others. The surface sections below carry lines for items this batch does not change -- those are not yours
- previous_price (decimal) -- the price this batch is changing the item AWAY from, read after 'was' on the item's line in Items In Batch. Copy the digits exactly as printed
- batch_price (decimal) -- the price this batch is changing the item TO, read after 'to' on the item's line in Items In Batch. Copy the digits exactly as printed -- some are stated to THREE decimal places and the third decimal is not noise, it is what tells you a rounding difference from a real one
- shelf_price (decimal) -- the price recorded against this item in the Shelf-Edge State section, after 'tag price'. Return null where that section carries no line for this item at all, and null where the line reads 'tag price -- not stated'
- pos_price (decimal) -- the price recorded against this item in the Point Of Sale State section, after 'till price'. Return null where that section carries no line for this item, and null where the line reads 'till price -- not stated'
- digital_price (decimal) -- the price recorded against this item in the Digital Channel State section, after 'online price'. Return null where that section carries no line for this item, and null where the line reads 'online price -- not stated'
- shelf_confirmed_on (date) -- the date a store confirmed the shelf tag was reprinted, from the Shelf-Edge State line, as yyyy-mm-dd. Return null where the line reads 'tag reprint -- not confirmed on this batch', and null where there is no line for this item. A null here does NOT mean the tag price is wrong -- it means nobody has looked at the tag
- pos_activated_on (date) -- the date the till file activated this price, from the Point Of Sale State line, as yyyy-mm-dd. Return null where there is no line for this item, and null where the line does not state one
- digital_published_on (date) -- the date the digital channel published this price, from the Digital Channel State line, as yyyy-mm-dd. Return null where there is no line for this item, and null where the line does not state one
- promotion_scope (enum) one of: none, digital, in_store -- what the Channel Promotions On File section records for this item: 'digital' where it authorises a digital-only promotional price, 'in_store' where it authorises an in-store one, and 'none' where that section carries no line for this item. Read this from the promotions section only
- promotion_price (decimal) -- the promotional price the Channel Promotions On File section authorises for this item. Return null where that section carries no line for it
- worst_gap_minor (derived) -- the LARGEST absolute difference, in MINOR UNITS (hundredths), between any two of the three recorded surface prices -- shelf against till, till against digital, shelf against digital. NO BATCH RECORD STATES IT; work it out. An integer, so a difference of 0.15 is 15. Return 0 where all three agree, and null where any one of the three surface prices is not recorded. It is measured against each other, NOT against the batch price
- divergence (enum) one of: aligned, rounding_difference, authorised_channel_price, not_propagated, diverged, not_determinable -- this item's status under the shipped rulebook. Work through six gates IN ORDER and stop at the first that fires. (1) NOT DETERMINABLE: the batch price or any one of the three surface prices is not recorded, so no comparison can be run. (2) AUTHORISED CHANNEL PRICE: a promotion is on file with a scope other than none, every surface its scope names carries the promotional price, and every surface it does not name carries the batch price. (3) NOT PROPAGATED: at least one surface still carries the item's PREVIOUS price. (4) ROUNDING DIFFERENCE: the batch price has a non-zero third decimal and every surface price is within one minor unit of it. (5) DIVERGED: the three surface prices are not all equal, or they are all equal and that price is not the batch price. (6) Anything that survives all five is ALIGNED. Never decide it from the Batch Notes remark

Return a JSON object of exactly this shape:
{"batch_ref": ..., "effective_from": ..., "rulebook_id": ..., "batch_note": ..., "items": [{"item_ref": ..., "previous_price": ..., "batch_price": ..., "shelf_price": ..., "pos_price": ..., "digital_price": ..., "shelf_confirmed_on": ..., "pos_activated_on": ..., "digital_published_on": ..., "promotion_scope": ..., "promotion_price": ..., "worst_gap_minor": ..., "divergence": ...}, ...]}
Use null for any field the batch record does not state.

PRICE CHANGE BATCH RECORD
-------------------------
Batch
-----
BCH-6576-GC

Batch Effective From
--------------------
2026-05-06

Pricing Rulebook
----------------
PIR-1 -- illustrative, shipped with this kit; reproduces no real pricing rule

Items In Batch
--------------
prices are stated in the trading currency's major unit; a third decimal is a unit-derived price
SKU-80435  Paracetamol tablets, 16 pack         was 0.70      to 0.55
SKU-55481  Laundry liquid 1.5L                  was 4.97      to 5.24
SKU-13125  Smoked back bacon 300g               was 4.27      to 3.44
SKU-52292  AA batteries, 8 pack                 was 7.12      to 6.20
SKU-80799  Olive oil 500ml                      was 7.11      to 7.95
SKU-51593  Mixing bowl, 24cm                    was 4.21      to 3.44
SKU-14330  Orange juice, from concentrate, 1L   was 0.91      to 0.76
SKU-40423  Chilled oat drink 1L                 was 1.74      to 2.06

Shelf-Edge State
----------------
SKU-14330  tag price 0.76                     tag reprinted 2026-05-06
SKU-53486  tag price 4.97                     tag reprinted 2026-03-22
SKU-51593  tag price 3.44                     tag reprinted 2026-05-06
SKU-52292  tag price 7.12                     tag reprinted 2026-03-12
SKU-40423  tag price 2.06                     tag reprinted 2026-05-06
SKU-69728  tag price 10.77                    tag reprinted 2026-04-05
SKU-55481  tag price 5.24                     tag reprinted 2026-05-06
SKU-13125  tag price 4.27                     tag reprinted 2026-02-26
SKU-80435  tag price 0.45                     tag reprinted 2026-05-06

Point Of Sale State
-------------------
SKU-69728  till price 10.77                   activated 2026-04-05
SKU-13125  till price 3.44                    activated 2026-05-06
SKU-53486  till price 4.97                    activated 2026-03-22
SKU-80435  till price 0.45                    activated 2026-05-06
SKU-52292  till price 6.20                    activated 2026-05-06
SKU-55481  till price -- not stated           activated 2026-05-06
SKU-14330  till price 0.76                    activated 2026-05-06
SKU-40423  till price 2.06                    activated 2026-05-06
SKU-51593  till price 2.83                    activated 2026-05-06
SKU-80799  till price 7.95                    activated 2026-05-06

Digital Channel State
---------------------
SKU-51593  online price 3.44                  published 2026-05-06
SKU-53486  online price 3.81                  published 2026-03-22
SKU-69728  online price 10.77                 published 2026-04-05
SKU-80435  online price 0.55                  published 2026-05-06
SKU-52292  online price 6.20                  published 2026-05-06
SKU-13125  online price 3.44                  published 2026-05-06
SKU-80799  online price 7.95                  published 2026-05-06
SKU-40423  online price 2.06                  published 2026-05-06
SKU-55481  online price 5.24                  published 2026-05-06
SKU-14330  online price 0.91                  published 2026-03-03

Channel Promotions On File
--------------------------
SKU-53486  digital-only promotional price 3.81 authorised for this period
SKU-80435  in-store promotional price 0.45 authorised for this period

Batch Notes
-----------
Flagged this batch for a second pass -- the tag reprints looked uneven to me.

Raw responseThe raw response

{"batch_ref":"BCH-6576-GC","effective_from":"2026-05-06","rulebook_id":"PIR-1","batch_note":"Flagged this batch for a second pass -- the tag reprints looked uneven to me.","items":[{"item_ref":"SKU-80435","previous_price":0.70,"batch_price":0.55,"shelf_price":0.45,"pos_price":0.45,"digital_price":0.55,"shelf_confirmed_on":"2026-05-06","pos_activated_on":"2026-05-06","digital_published_on":"2026-05-06","promotion_scope":"in_store","promotion_price":0.45,"worst_gap_minor":10,"divergence":"authorised_channel_price"},{"item_ref":"SKU-55481","previous_price":4.97,"batch_price":5.24,"shelf_price":5.24,"pos_price":null,"digital_price":5.24,"shelf_confirmed_on":"2026-05-06","pos_activated_on":"2026-05-06","digital_published_on":"2026-05-06","promotion_scope":"none","promotion_price":null,"worst_gap_minor":null,"divergence":"not_determinable"},{"item_ref":"SKU-13125","previous_price":4.27,"batch_price":3.44,"shelf_price":4.27,"pos_price":3.44,"digital_price":3.44,"shelf_confirmed_on":"2026-02-26","pos_activated_on":"2026-05-06","digital_published_on":"2026-05-06","promotion_scope":"none","promotion_price":null,"worst_gap_minor":83,"divergence":"not_propagated"},{"item_ref":"SKU-52292","previous_price":7.12,"batch_price":6.20,"shelf_price":7.12,"pos_price":6.20,"digital_price":6.20,"shelf_confirmed_on":"2026-03-12","pos_activated_on":"2026-05-06","digital_published_on":"2026-05-06","promotion_scope":"none","promotion_price":null,"worst_gap_minor":92,"divergence":"not_propagated"},{"item_ref":"SKU-80799","previous_price":7.11,"batch_price":7.95,"shelf_price":null,"pos_price":7.95,"digital_price":7.95,"shelf_confirmed_on":null,"pos_activated_on":"2026-05-06","digital_published_on":"2026-05-06","promotion_scope":"none","promotion_price":null,"worst_gap_minor":null,"divergence":"not_determinable"},{"item_ref":"SKU-51593","previous_price":4.21,"batch_price":3.44,"shelf_price":3.44,"pos_price":2.83,"digital_price":3.44,"shelf_confirmed_on":"2026-05-06","pos_activated_on":"2026-05-06","digital_published_on":"2026-05-06","promotion_scope":"none","promotion_price":null,"worst_gap_minor":61,"divergence":"diverged"},{"item_ref":"SKU-14330","previous_price":0.91,"batch_price":0.76,"shelf_price":0.76,"pos_price":0.76,"digital_price":0.91,"shelf_confirmed_on":"2026-05-06","pos_activated_on":"2026-05-06","digital_published_on":"2026-03-03","promotion_scope":"none","promotion_price":null,"worst_gap_minor":15,"divergence":"not_propagated"},{"item_ref":"SKU-40423","previous_price":1.74,"batch_price":2.06,"shelf_price":2.06,"pos_price":2.06,"digital_price":2.06,"shelf_confirmed_on":"2026-05-06","pos_activated_on":"2026-05-06","digital_published_on":"2026-05-06","promotion_scope":"none","promotion_price":null,"worst_gap_minor":0,"divergence":"aligned"}]}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch a store's price tag that disagrees with itself — 50 price-change batches. One model answered, and every answer was then graded Six different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid -- the real spend is in the shared call ledger, not on this page.

The gradersSix ways to grade

The floor is scored by the SAME judge, over the same gold, on the same 50 records. A floor scored by a different scorer from the run it is compared against is not a floor, it is a second kit.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — top_k, chunk size, prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The free system-level floor (evals/baseline.py), the gold gate (evals/check_labels.py) and the scorer (evals/judge.py) are all pure code and cost $0.00 to run against any result set. The figure above is the ONE scored run's own token counts (50 records) priced at Google Gemini 3 Flash's published rate -- the same basis cost_per_query_usd uses. ⚠︎ IT IS NOT WHAT BUILDING THIS KIT COST. Outside the scored run: a 4-record calibration at max_tokens=40000 fired at the four LONGEST batches before anything was scored ($0.1685, and it earned its money -- see below), the worked-example capture ($0.0208), 4 prompt-split calls at max_tokens=1 ($0.0062), and 1 live UI screenshot. Sixty live calls sit behind this page in total, projecting to roughly $1.16 on this card. ⚑ AND THE CALIBRATION IS THE LINE ITEM TO READ, BECAUSE IT CHANGED THE CODE. It measured a largest reply of 34922 output tokens against a ceiling this file had been written with at 32,000 -- so $0.1685 of calibration bought the knowledge that the published constant would have cut a batch. It was raised to 96,000 BEFORE the scored run. A calibration that confirms your guess is cheap; this one did not.

Cost driversWhat actually moves the bill

• PROVIDER-SIDE REASONING, by a distance. 228915 of 276922 output tokens (82.7 pct) never reach the reply text and are billed in full. The JSON roster itself is the other 17.3 pct.
• THE FIXED PREFIX, which is the same on every call. 3761 of 4794 input tokens (78 pct) are the system prompt, the rulebook and the field schema; the batch record itself is 1033. Nothing here caches it.
• THE ITEM COUNT, which drives the output. The reply is one twelve-field object per item, so a nine-item batch costs meaningfully more than a five-item one -- and the corpus averages 6.6.
• THE FIELD SCHEMA, which is the most expensive single part of the prompt at 1282 tokens -- more than the rulebook. Twelve per-item fields each need a hint saying WHICH of three identically-formatted sections a value comes from.

Your volumeWhat it costs at your volume

Linear in batches and NOT linear in batch size. Ten times the batches is ten times the bill: each is one independent call with an identical prefix. But ten times the ITEMS PER BATCH is worse than ten times the cost, because the reply is a roster -- output grows with the item count while the prefix stays fixed, so the per-item cost falls and the per-CALL cost rises until it stops fitting the ceiling. This corpus's largest batch is 9 items and its largest reply was 13183 output tokens; the shape has to change long before a real supermarket batch of thousands, and no measurement here says where.

Model choiceWhy this model

It is the tier the shared key on this machine is pointed at, and one tier is what this run bought. The kit is provider-agnostic -- src/adapters reaches an OpenAI-compatible endpoint or Anthropic's Messages API over urllib -- so 'why this model' is honestly 'it is the one configured', and every figure here would move on another.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 batch records, one completion call each, one tier. The calibration, the worked-example capture, the prompt-split calls and the UI screenshot are recorded separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

10 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pyBatch record segmenter

cuts one extract into addressable sections on its underlined headings; falls back to one whole-document segment when it finds none

# Cut a batch record into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pySection selector — a swap seam

maps each of the 17 fields to the sections that can carry it, and subtracts the two sections no field maps to -- unconditionally, including on the fallback path

You change it to: which sections reach the provider at all. NEVER_SENT is the privacy line; _fallback subtracts it unconditionally so an absent section cannot widen the wire.

# Pick which sections of a batch record are sent. Pure code -- the last deterministic step before

BATCH = "Batch"
EFFECTIVE = "Batch Effective From"
RULEBOOK = "Pricing Rulebook"
ITEMS = "Items In Batch"
SHELF = "Shelf-Edge State"
POS = "Point Of Sale State"
DIGITAL = "Digital Channel State"
PROMOS = "Channel Promotions On File"
NOTES = "Batch Notes"

src/prompt.pyPrompt assembler

one call per batch: the system prompt, the rulebook rendered from its own JSON, the field schema, and the mapped sections

# Assemble the price-divergence prompt. ONE prompt per batch record, every item on it in one

VERDICTS = ("aligned", "rounding_difference", "authorised_channel_price",
VERDICT_MEANINGS = {
SYSTEM = (
def rulebook_block(rb=None):
RULEBOOK_TEXT = rulebook_block()
def field_schema(fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/adapters/__init__.pyProvider adapter

urllib against an OpenAI-compatible endpoint or Anthropic's Messages API, with backoff over both HTTP and transport failures

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/rulebook.pyThe pricing rulebook

the six gates, in order, as pure code over integer thousandths -- one definition read by the corpus generator, the prompt and the scorer

# THE PRICING-INTEGRITY RULEBOOK, loaded from data/rulebook.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
STATUSES = ("aligned", "rounding_difference", "authorised_channel_price",
EXCEPTIONS = ("not_propagated", "diverged")
NOTHING_TO_DO = ("aligned", "rounding_difference", "authorised_channel_price")
SURFACES = ("shelf", "pos", "digital")
SURFACE_FIELD = {"shelf": "shelf_price", "pos": "pos_price", "digital": "digital_price"}
PROMOTION_SCOPES = ("none", "digital", "in_store")
PROPAGATION_STATES = ("complete", "partial", "none")

src/extract.pyExtractor — a swap seam

drives the whole path and attaches a span to every spannable value, searched inside the section the field is mapped to before it ever widens

You change it to: MAX_TOKENS. A cut reply costs a whole batch and produces no exception list at all, which looks exactly like a batch with nothing wrong.

# Extract one price-change batch record: segment, select, prompt, ONE model call, then the

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 96000
def load_fields():
def item_key(fields):
def scored_item_fields(fields):
def load_doc(batch_id):
def documents():

src/extract.pyBatch routing rule — a swap seam

the pure-code business condition -- does this batch need a price decided or a job re-run

You change it to: MAX_TOKENS. A cut reply costs a whole batch and produces no exception list at all, which looks exactly like a batch with nothing wrong.

# Extract one price-change batch record: segment, select, prompt, ONE model call, then the

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 96000
def load_fields():
def item_key(fields):
def scored_item_fields(fields):
def load_doc(batch_id):
def documents():

evals/baseline.pyThe free floor

the system-level check: till and digital against the batch, blind to the shelf edge by construction

# A free, rules-and-regex monitor. No model, no key, no spend -- scored by the same judge as a

NOT_STATED = "-- not stated"
NOT_CONFIRMED = "tag reprint -- not confirmed on this batch"
SURFACE_SECTION = {"shelf": "Shelf-Edge State", "pos": "Point Of Sale State",
SURFACE_LABEL = {"shelf": "tag price", "pos": "till price", "digital": "online price"}
STAMP_LABEL = {"shelf": "tag reprinted", "pos": "activated", "digital": "published"}
def _section(text, name):
def _lines(text, name):
def _cols(line):
def _roster(text):

evals/judge.pyScorer

six graders, pure code, no LLM judge

# Score a price-divergence run. PURE CODE -- gold is exact and every answer is one value, so `==`

def norm(v):
def equal(field, got, want):
def _rows_by_ref(items, key="item_ref"):
def score(fields, records, golds):
def _matrix(rows, positive):
NOTHING_TO_DO = RB.NOTHING_TO_DO
def is_shelf_only(gp):
def score_statuses(fields, records, golds):
def _same_int(got, want):

src/app.pyLocal UI

one batch, one call, on 127.0.0.1:8975; renders with no key

# The minimal local UI. Standard library only -- python -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "8975"))
class H(BaseHTTPRequestHandler):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 4672 input and 5538 output tokens per query at top-k 1, of which about 150 tokens are fixed (system prompt, chat framing, the question itself) and the rest is retrieved context. Change the retrieval depth and the input scales with it. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model and top-k follow the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's batch records are entirely synthetic (tools/build_corpus.py, seed 20260822): no real retailer, store, product, brand, person, price or currency exists in the corpus, and nothing was fetched from anywhere. No statute, regulation, weights-and-measures rule, state's adopted text or retailer pricing policy is reproduced anywhere in the kit. The only outbound traffic the kit makes is one chat-completion request per batch record, carrying the mapped sections plus the fixed prompt and the shipped rulebook -- and the two sections a real price-control desk would most mind sending anywhere, Store Contact (the store number, its price-control lead by name, a direct telephone extension and an internal store account reference) and Prepared By, are mapped by no field and never leave the machine. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser.

The experimentWe did not attack it -- and the privacy boundary is guarded by a check, not by luck

Three boundaries hold. Two were confirmed by reading the code: no code path writes a price, re-sends a batch or writes anything at all, and the rulebook reads four prices and two promotion fields off a row it is handed, so no record text can reach the decision. The third is confirmed by a CHECK RATHER THAN BY READING, and it had to be -- the fallback that would widen the wire to the store contact block never fires on this corpus, so evals/check_labels.py calls _fallback and for_field directly, on every document and every field plus a field name no hint matches, and asserts the two never-sent sections are absent from all of it. A guard that holds because nothing ever reaches it is not a guard. The fourth is open: an indirect prompt injection needs a section an outside party authored, and a store's own shelf-edge return is exactly that. Confirmed by reading the code and by a pre-flight assertion that drives the fallback path directly, not by an attack run, on 2026-08-22 -- no attack run was fired; see redteam.why.

Three boundaries hold and the second is the one worth reading, because it holds by a check rather than by luck: the fallback path that would leak the store contact never fires on this corpus, so it is driven directly by the gold gate on every document before any run may spend. The fourth is open and is named as open -- a price extract is assembled from several systems and a store, which makes an instruction hidden in a note exactly the attack this kit's shape invites, and nothing here has tried it.

The result0 attack trials. Three of four boundaries checked here hold on evidence -- and the privacy one holds because a gate drives a code path this corpus never reaches, not because the path was never taken. The one that does not hold is the one this kit's shape invites: a price extract is assembled by several systems and a store, and whether an instruction hidden in a note could make an item read aligned is unmeasured.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted decoys measure whether the run reports the batch's own items and no others, whether it separates an authorised channel price from a divergence, and whether it declines an item the extract cannot answer for. None of those measures whether it obeys an instruction hidden in the same file, and on this kit the cheap attack is not to change a call -- it is to make one item read aligned.

HonestyWhat this does not prove

• Whether an indirect prompt injection in a shelf-edge return or a Batch Notes line could move a call. No attack run exists, and the make-it-read-aligned version is the one this kit's own shape suggests would be cheapest, because a cleared item is an item nobody looks at again.
• Whether the kit behaves safely against a hostile provider -- a response body crafted to break the roster parsing in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed record, and nothing beyond that was tested.
• Whether a real price extract would carry anything sensitive this kit mishandles beyond the store contact block. The corpus has no personal data by construction; a real extract carries margin, cost price, competitor-match flags and named colleagues, and none of that path is exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_price_decision fires when the batch carries at least one item whose computed status is `diverged`, or at least one whose status is `not_determinable`. Both values come out of the same reply; the rule is run afterwards in pure code, over whatever the model returned, and never over gold. A reply carrying no item the rule can read returns None rather than False -- an unknown is not a pass.

src/extract.py::compute(), called by extract() on every record, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by identical code. The six-gate rule it reads lives in a SEPARATE module, src/rulebook.py, which is also what the corpus generator wrote gold with and what src/prompt.py states to the model -- one definition, three readers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a correction, a price push, a batch retrigger or a decision about what any price should be. Nothing here changes anything. divergence is a finding with the rulebook's own reasoning attached, and a pricing-integrity desk decides.
• It is NOT a check that the exception list is COMPLETE, and the corpus is the demonstration. The rule reads the reply; if the reply misses an item, the rule is silent about it. On this run the reply missed nothing, so the flag was perfect -- which measures the run and not the guardrail. A guardrail downstream of a reading step inherits that step's blind spots entirely.
• It is NOT any retailer's escalation policy. 'A divergence or an unreadable record needs a person; a failed propagation needs a retry' is this kit's own simplification, invented for this corpus. No retailer's procedure, law or regulation was consulted, and none is reproduced. A real desk weighs how many customers have already been through the till, how large the gap is and in which direction, and what the store can physically do today.
• It is NOT a judgement about the rulebook. The flag reads the statuses the rulebook produced; if the rulebook is wrong, the flag is confidently wrong with it. The shipped rulebook is illustrative and nobody qualified has reviewed it.
• ⚠︎ AND IT IS NOT A SHELF INSPECTION. An item cleared as aligned on a tag no store has confirmed is cleared on the pricing system's own record of what it SENT to the tag printer. 43 of the 332 item rows are in that state and 26 were cleared. The count is reported beside every exception list, in the UI and in the run record, and it is not part of this flag.

WatchedWhat is watched, and why that one

The kit measures 24 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 17 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 50, extraction_cells 4184, failures 0, refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

2 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 2 runs measured the same system — they differ on provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 2 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/extract.py, on any change to data/rulebook.json (which moves the statuses the flag reads), on any change to data/fields.json's allowed values for divergence or promotion_scope, and on any corpus regeneration -- the gate re-checks that the flag is not constant every time. Changing WHAT GETS ESCALATED is a policy change and must be re-scored, even though it never changes what the rulebook says.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the rulebook is a JSON file read with json, money is held as integer thousandths rather than through a decimal library, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

One path per record -- segment, select, prompt, one call, parse, span, the rule -- with no branch, no loop and no agent. What the reply contains is a ROSTER rather than one object, and that is the only structural difference from a flat-record extraction kit: the reply's length tracks the item count, so the output ceiling is a real constraint and the scorer has to align rows by key before it can compare a single cell.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain -- the JSON extraction from a fenced reply, the roster parsing, the backoff policy, the section regex, the money parser and the .env reader are all code somebody has to own. ⚠︎ AND THIS KIT PAID THAT BILL IN PUBLIC: its ONLY error surface on the scored run is json.loads silently turning the model's 6.20 into 6.2, which cost 142 of 4184 cells and 142 spans. A schema-validating library with a string type on those fields would have caught it; hand-rolled parsing did not.
• No framework means no framework's ecosystem -- no tracing, no eval harness beyond the one in evals/, no prompt registry, no structured-output enforcement. The reply is a ROSTER and nothing validates its shape except src/prompt.py::parse, which fails closed to ({}, []) rather than to an empty batch, because 'no exceptions on this batch' is exactly the wrong thing to say by accident.
• The corpus generator, the gold gate and the scorer are three separate programs that must agree, and only convention holds them together -- one shared import of src/rulebook.py. It works, and it is why every published label is the rule's own output rather than a typed value; it is also three places to change when the rule changes.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-price-divergence on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 1 earlier run on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 1 run on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT (this repository's own licence). Nothing was fetched and nothing is licensed from anybody: no real retailer, store, product, brand, person or currency is named anywhere in the kit. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-(item, field) exact match, light normalisation

Does the model's value for each of the 12 per-item fields match gold after trimming whitespace and punctuation, plus four batch-level cells per record? THE DENOMINATOR IS GOLD'S ITEMS, NOT THE RUN'S: an item the run never returned would score as a miss on all twelve of its cells rather than disappearing. It is the reference grader because every other number on this page is a consequence of these cells being right.

MethodHow the test was run

evals/judge.py, in-process, no key and no model. The same function evals/run.py calls after every run, and the same logic the free floor is scored by -- a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

Read the numbers rather than the verdicts. The model called this row perfectly and still lost three cells: the record prints 6.20, the model wrote 6.20 on the wire, and json.loads handed the scorer the float 6.2. The rule never noticed -- src/rulebook.py::milli normalises to integer thousandths -- so the status, the gap and the surface at issue are all right while per-cell exact match convicts the row. The same three values also ship with no span, because a span is searched for AS PRINTED. That is this kit's entire error surface, on one row.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, derived by running src/rulebook.py over the same values each record states; evals/check_labels.py re-runs the rule over all 332 rows and asserts every hard case has a floor under it before any run is allowed to spend.

How tight can the band be? No tolerance band anywhere. Exact match after trimming whitespace and punctuation for cells; integer comparison for the gap; a closed set of six values for the status. Nothing here is a continuous score to round.

Cadence: Re-score on any change to data/rulebook.json, data/fields.json, src/prompt.py or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe divergence call, six-way, over the model's own values

Does src/rulebook.py, run over the values the MODEL returned, land on gold's status? This is the deciding half and it is not a tautology: a misread price flips a status, so it measures the whole pipeline rather than the rule.

MethodHow the test was run

evals/judge.py, in-process, no key and no model. The same function evals/run.py calls after every run, and the same logic the free floor is scored by -- a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

Read the numbers rather than the verdicts. The model called this row perfectly and still lost three cells: the record prints 6.20, the model wrote 6.20 on the wire, and json.loads handed the scorer the float 6.2. The rule never noticed -- src/rulebook.py::milli normalises to integer thousandths -- so the status, the gap and the surface at issue are all right while per-cell exact match convicts the row. The same three values also ship with no span, because a span is searched for AS PRINTED. That is this kit's entire error surface, on one row.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, derived by running src/rulebook.py over the same values each record states; evals/check_labels.py re-runs the rule over all 332 rows and asserts every hard case has a floor under it before any run is allowed to spend.

How tight can the band be? No tolerance band anywhere. Exact match after trimming whitespace and punctuation for cells; integer comparison for the gap; a closed set of six values for the status. Nothing here is a continuous score to round.

Cadence: Re-score on any change to data/rulebook.json, data/fields.json, src/prompt.py or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe gap arithmetic, the model's own, which decides nothing

The largest pairwise difference between the three recorded surface prices, in minor units, measured against EACH OTHER and not against the batch price. No record states it. The kit publishes the CODE's figure, which is exactly what makes this a clean measurement of arithmetic rather than of copying.

MethodHow the test was run

evals/judge.py, in-process, no key and no model. The same function evals/run.py calls after every run, and the same logic the free floor is scored by -- a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

Read the numbers rather than the verdicts. The model called this row perfectly and still lost three cells: the record prints 6.20, the model wrote 6.20 on the wire, and json.loads handed the scorer the float 6.2. The rule never noticed -- src/rulebook.py::milli normalises to integer thousandths -- so the status, the gap and the surface at issue are all right while per-cell exact match convicts the row. The same three values also ship with no span, because a span is searched for AS PRINTED. That is this kit's entire error surface, on one row.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, derived by running src/rulebook.py over the same values each record states; evals/check_labels.py re-runs the rule over all 332 rows and asserts every hard case has a floor under it before any run is allowed to spend.

How tight can the band be? No tolerance band anywhere. Exact match after trimming whitespace and punctuation for cells; integer comparison for the gap; a closed set of six values for the status. Nothing here is a continuous score to round.

Cadence: Re-score on any change to data/rulebook.json, data/fields.json, src/prompt.py or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineShelf-only failures, the population a system-level check cannot see

Of the items where the TILL and the DIGITAL channel both took the new price and the physical tag did not, how many reached the exception list at all -- and how many were additionally given the right one of the two exception statuses? The population is derived from gold's own values in evals/judge.py::is_shelf_only, never from a bucket name the gold file does not carry.

MethodHow the test was run

evals/judge.py, in-process, no key and no model. The same function evals/run.py calls after every run, and the same logic the free floor is scored by -- a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

Read the numbers rather than the verdicts. The model called this row perfectly and still lost three cells: the record prints 6.20, the model wrote 6.20 on the wire, and json.loads handed the scorer the float 6.2. The rule never noticed -- src/rulebook.py::milli normalises to integer thousandths -- so the status, the gap and the surface at issue are all right while per-cell exact match convicts the row. The same three values also ship with no span, because a span is searched for AS PRINTED. That is this kit's entire error surface, on one row.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, derived by running src/rulebook.py over the same values each record states; evals/check_labels.py re-runs the rule over all 332 rows and asserts every hard case has a floor under it before any run is allowed to spend.

How tight can the band be? No tolerance band anywhere. Exact match after trimming whitespace and punctuation for cells; integer comparison for the gap; a closed set of six values for the status. Nothing here is a continuous score to round.

Cadence: Re-score on any change to data/rulebook.json, data/fields.json, src/prompt.py or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe false-alarm rate on the exception list

Of every item this kit put on the exception list, what share did not belong there? Counted STRICTLY: a false alarm is a flagged row whose gold status is aligned, rounding_difference or authorised_channel_price -- the three answers that mean nothing needed doing. A flagged not_determinable row is counted separately as MISROUTED, because somebody does have to open that record; for a different reason.

MethodHow the test was run

evals/judge.py, in-process, no key and no model. The same function evals/run.py calls after every run, and the same logic the free floor is scored by -- a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

Read the numbers rather than the verdicts. The model called this row perfectly and still lost three cells: the record prints 6.20, the model wrote 6.20 on the wire, and json.loads handed the scorer the float 6.2. The rule never noticed -- src/rulebook.py::milli normalises to integer thousandths -- so the status, the gap and the surface at issue are all right while per-cell exact match convicts the row. The same three values also ship with no span, because a span is searched for AS PRINTED. That is this kit's entire error surface, on one row.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, derived by running src/rulebook.py over the same values each record states; evals/check_labels.py re-runs the rule over all 332 rows and asserts every hard case has a floor under it before any run is allowed to spend.

How tight can the band be? No tolerance band anywhere. Exact match after trimming whitespace and punctuation for cells; integer comparison for the gap; a closed set of six values for the status. Nothing here is a continuous score to round.

Cadence: Re-score on any change to data/rulebook.json, data/fields.json, src/prompt.py or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe batch routing flag against the same rule run over gold

src/extract.py::compute() over the run's OWN computed statuses, against the same rule run over gold's: does this batch need somebody to DECIDE a price, or somebody to re-run a job? A business condition, so unlike the consistency diagnostic it genuinely needs labels, and saying so is half of what makes the number believable.

MethodHow the test was run

evals/judge.py, in-process, no key and no model. The same function evals/run.py calls after every run, and the same logic the free floor is scored by -- a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

Read the numbers rather than the verdicts. The model called this row perfectly and still lost three cells: the record prints 6.20, the model wrote 6.20 on the wire, and json.loads handed the scorer the float 6.2. The rule never noticed -- src/rulebook.py::milli normalises to integer thousandths -- so the status, the gap and the surface at issue are all right while per-cell exact match convicts the row. The same three values also ship with no span, because a span is searched for AS PRINTED. That is this kit's entire error surface, on one row.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, derived by running src/rulebook.py over the same values each record states; evals/check_labels.py re-runs the rule over all 332 rows and asserts every hard case has a floor under it before any run is allowed to spend.

How tight can the band be? No tolerance band anywhere. Exact match after trimming whitespace and punctuation for cells; integer comparison for the gap; a closed set of six values for the status. Nothing here is a continuous score to round.

Cadence: Re-score on any change to data/rulebook.json, data/fields.json, src/prompt.py or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it