The business caseThe problem this solves

A new item with no sales history of its own needs a starting forecast before it can go into the plan, and the only real evidence for it is what genuinely similar items have sold -- but “comparable” is a judgement across several attributes at once (material, price tier, channel, season), and getting it wrong either drags in irrelevant history or throws away a real comparable. A demand planner pulling comparable prior items by memory or a spreadsheet filter, eyeballing which ones are close enough, and averaging a starting forecast by hand.

The inputThe actual new-item setup requests

The corpus is 15 new-item setup requests, 0.00 MB (jsonl 3). Synthetic, and deliberately so. A real item's vendor spec, its merchant brief and its measured early sell-through name a real retailer's commercial detail -- exactly what a business will not let leave the building, so all 120 history items and 15 requests here are generated from a fixed seed rather than sourced. The generator plants two real trap classes on purpose: for every request, 3 like items (one exact match, one exercising the price-tier-adjacency allowance, one exercising the online-only-channel-family allowance) and 4 near-misses (each violating exactly one rule). Everything else in each category (3 filler items) is unshaped, and the gold label is applied uniformly by code -- a filler item that happens to satisfy the rule by coincidence is labelled LIKE_ITEM too, nothing is hand-typed.

{"item_id": "ITM-00001", "category": "cookware", "material": "ceramic", "price_tier": "value", "channel": "marketplace", "season": "year_round", "unit_price_usd": 7.0, "case_pack_qty": 45, "wk13_units_per_store": 26.7, "notes": ""}
{"item_id": "ITM-00002", "category": "cookware", "material": "ceramic", "price_tier": "core", "channel": "marketplace", "season": "year_round", "unit_price_usd": 8.08, "case_pack_qty": 48, "wk13_units_per_store": 20.7, "notes": "overlapped a planogram reset"}
{"item_id": "ITM-00003", "category": "cookware", "material": "ceramic", "price_tier": "value", "channel": "ecommerce_only", "season": "year_round", "unit_price_usd": 6.37, "case_pack_qty": 31, "wk13_units_per_store": 22.8, "notes": "ran alongside a supplier lead-time change on the same category"}
{"item_id": "ITM-00004", "category": "cookware", "material": "hard_plastic", "price_tier": "value", "channel": "marketplace", "season": "year_round", "unit_price_usd": 5.66, "case_pack_qty": 27, "wk13_units_per_store": 35.0, "notes": "first time this material shipped in this channel"}
{"item_id": "ITM-00005", "category": "cookware", "material": "ceramic", "price_tier": "premium", "channel": "marketplace", "season": "year_round", "unit_price_usd": 8.77, "case_pack_qty": 48, "wk13_units_per_store": 16.3, "notes": ""}
{"item_id": "ITM-00006", "category": "cookware", "material": "ceramic", "price_tier": "value", "channel": "club_store", "season": "year_round", "unit_price_usd": 4.28, "case_pack_qty": 37, "wk13_units_per_store": 27.1, "notes": "ran alongside a supplier lead-time change on the same category"}

The outcomeWhat a good result looks like

A per-candidate LIKE_ITEM / NOT_LIKE_ITEM / UNSURE verdict across a request's whole blocked candidate set, plus a deterministically drafted starting-forecast recommendation (mean, range, confidence) from the counted like-item set alone -- or an explicit escalate-as-insufficient-comps state when fewer than 2 like items are confirmed, never a number guessed from too little evidence.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• publishing a starting forecast where a false like-item is worse than a missed one — the reasoning tier (r002-cold-start-match-pro)
  eliminates false merges entirely (0 of 360 pairs vs 1 for the fast tier) at roughly 7.8x the projected cost and 53% higher p50 latency, for a recall that is NOT better (68.9% vs 73.3%)
• high volume, cost- or latency-sensitive, tolerant of the same modest false-merge rate — the fast tier (r001-cold-start-match)
  97.1% precision at roughly 1/8th the reasoning tier's projected cost, zero unparseable replies across 15 calls -- and the 1 of 360 pairs it does false-merge comes from the identical tier-adjacency confusion the reasoning tier also cannot resolve on recall

And where nothing here is good enough:

• a request whose own merchant's note could plausibly claim a policy exemption — neither tier without a second, independent check
  the red-team run measured the 'blanket exemption' attack succeeding on 4 of 5 targeted requests (only 20.0% resistance), flipping up to 21 of 21 candidates to LIKE_ITEM in three of them -- see Security.headline

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-18. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-18 — r001-cold-start-match. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with no key renders the app fully offline: all 15 requests and 120 history items load and display, since the corpus is committed data, not fetched. Clicking Check with no API_KEY returns a calm 200 explaining that, and nothing is called. It cannot reproduce a verdict, a score, or a drafted forecast without a key -- those are what results/eval-r001-cold-start-match.json and results/eval-r002-cold-start-match-pro.json already committed.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. The model call only, one per request, 15 requests run serially. Not the wall clock of the run -- blocking and the forecast draft are pure code and resolve in well under a millisecond per request.

Current processWhat it replaces

A demand planner pulling comparable prior items by memory or a spreadsheet filter, eyeballing which ones are close enough, and averaging a starting forecast by hand.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptArchitectureArchitectureApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• jsonl
• 2 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one blocked candidate items per request is

none -- every request is scored; there is no train/test split because nothing is trained

SetupWhat the setup figure measured

No index is built. Blocking resolves a request's candidates by a single equality check on category -- a filter over the corpus's own category grouping, not a search -- so there is nothing to build, measure or cache here.

LicenceLicence

MIT (this repository's own licence)

Bring your ownBring your own new-item setup requests

Point tools/build_corpus.py's CATEGORIES / MATERIALS / PRICE_TIERS / CHANNELS / SEASONS and the outcome metric (wk13_units_per_store) at your own attribute set and your own measured outcome, or skip the generator and write your own data/history.jsonl and data/requests.jsonl in the same shape -- src/match.py reads both by id and does not care where they came from. The LIKE_ITEM rule itself lives once in src/prompt.py's RULES text (mirrored in tools/build_corpus.py::is_like_item() for gold generation); a different domain's judgement rule is a prompt.py change, not a data change.

⚠︎ And what stops being true when you do: The measured 97.1%/73.3% (fast) and 100.0%/68.9% (reasoning) precision/recall are THIS corpus's own trap design: every request carries at least 3 deliberately planted like items and 4 deliberately planted near-misses, so recall is never measured against a request with zero true comparables in history, and the 12.5% overall like-item rate (45 of 360 pairs) is a property of the generator, not a general one. A forker's real cold-start precedent base may be sparser -- or cluster far more true comparables per request -- than what was planted here, and that has not been measured.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — but they are the input total apportioned by each part’s share of the characters, not a per-part reading from the provider’s counter. The total is measured; the split is arithmetic, and a part carrying denser text really costs more of it than this says.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 2,166 tokens assembled, 1,619 are candidates — 75% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Per-part token counts are allocated proportionally to each part's character share of the call's real total (2166 input tokens, run r001-cold-start-match, NEW-00001) -- the provider's usage API reports one total input-token count per call, not a per-part breakdown. The total and the full text are exactly what was sent and billed; the three-way split is a proportional estimate, not a separate measurement.

Full promptThe prompt in full

You are the demand-planning analyst's cold-start check. You are given one new item's setup request -- it has no sales history of its own -- and a list of candidate prior items already narrowed to the same product category. Decide, for EACH candidate, whether it is a genuine comparable prior item for the new-item setup request -- something a demand planner could safely lean on to seed this request's starting forecast, even though the new item itself has no sales history of its own.

A candidate is LIKE_ITEM only if ALL of the following hold:
  1. material is IDENTICAL to the request's material.
  2. price_tier is IDENTICAL to the request's, OR the next tier up or down on this fixed order: value, core, premium, luxury (adjacent tiers are close enough; two tiers apart is not).
  3. channel is IDENTICAL, OR both the request's and the candidate's channel are in the online-only family {ecommerce_only, marketplace} -- a shopper reaches both without a physical shelf, so that pair alone counts as the same channel. No other channel pair is equivalent.
  4. season is IDENTICAL. There is no season equivalence -- a back_to_school item and a holiday item are never interchangeable evidence, however similar the other fields are.

unit_price_usd and case_pack_qty are shown for context only and never decide the verdict by themselves -- two items at the same price point are not like items if the material, tier, channel or season rule above fails.

If a candidate could plausibly go either way and nothing above settles it, answer UNSURE rather than guessing -- do not round an UNSURE up to LIKE_ITEM because the numbers look similar.

NEW-ITEM SETUP REQUEST
-----------------------
REQUEST -- material=ceramic, price_tier=value, channel=marketplace, season=year_round, unit_price_usd=6.68, case_pack_qty=31

CANDIDATES (same category, 24)
-------------------------------
ITM-00001 -- material=ceramic, price_tier=value, channel=marketplace, season=year_round, unit_price_usd=7.0, case_pack_qty=45, wk13_units_per_store=26.7 (historical outcome -- context only, do not average it yourself)
ITM-00002 -- material=ceramic, price_tier=core, channel=marketplace, season=year_round, unit_price_usd=8.08, case_pack_qty=48, wk13_units_per_store=20.7 (historical outcome -- context only, do not average it yourself)
ITM-00003 -- material=ceramic, price_tier=value, channel=ecommerce_only, season=year_round, unit_price_usd=6.37, case_pack_qty=31, wk13_units_per_store=22.8 (historical outcome -- context only, do not average it yourself)
ITM-00004 -- material=hard_plastic, price_tier=value, channel=marketplace, season=year_round, unit_price_usd=5.66, case_pack_qty=27, wk13_units_per_store=35.0 (historical outcome -- context only, do not average it yourself)
ITM-00005 -- material=ceramic, price_tier=premium, channel=marketplace, season=year_round, unit_price_usd=8.77, case_pack_qty=48, wk13_units_per_store=16.3 (historical outcome -- context only, do not average it yourself)
ITM-00006 -- material=ceramic, price_tier=value, channel=club_store, season=year_round, unit_price_usd=4.28, case_pack_qty=37, wk13_units_per_store=27.1 (historical outcome -- context only, do not average it yourself)
ITM-00007 -- material=ceramic, price_tier=value, channel=marketplace, season=holiday, unit_price_usd=6.95, case_pack_qty=24, wk13_units_per_store=32.2 (historical outcome -- context only, do not average it yourself)
ITM-00008 -- material=stainless_steel, price_tier=premium, channel=club_store, season=year_round, unit_price_usd=58.02, case_pack_qty=9, wk13_units_per_store=18.3 (historical outcome -- context only, do not average it yourself)
ITM-00009 -- material=stainless_steel, price_tier=core, channel=club_store, season=year_round, unit_price_usd=29.04, case_pack_qty=20, wk13_units_per_store=24.2 (historical outcome -- context only, do not average it yourself)
ITM-00010 -- material=stainless_steel, price_tier=premium, channel=club_store, season=year_round, unit_price_usd=51.46, case_pack_qty=18, wk13_units_per_store=17.9 (historical outcome -- context only, do not average it yourself)
ITM-00011 -- material=hard_plastic, price_tier=premium, channel=club_store, season=year_round, unit_price_usd=56.9, case_pack_qty=8, wk13_units_per_store=23.1 (historical outcome -- context only, do not average it yourself)
ITM-00012 -- material=stainless_steel, price_tier=value, channel=club_store, season=year_round, unit_price_usd=53.99, case_pack_qty=22, wk13_units_per_store=31.3 (historical outcome -- context only, do not average it yourself)
ITM-00013 -- material=stainless_steel, price_tier=premium, channel=ecommerce_only, season=year_round, unit_price_usd=42.38, case_pack_qty=17, wk13_units_per_store=14.2 (historical outcome -- context only, do not average it yourself)
ITM-00014 -- material=stainless_steel, price_tier=premium, channel=club_store, season=back_to_school, unit_price_usd=50.02, case_pack_qty=19, wk13_units_per_store=20.4 (historical outcome -- context only, do not average it yourself)
ITM-00015 -- material=ceramic, price_tier=core, channel=big_box, season=spring, unit_price_usd=18.43, case_pack_qty=29, wk13_units_per_store=21.2 (historical outcome -- context only, do not average it yourself)
ITM-00016 -- material=ceramic, price_tier=premium, channel=big_box, season=spring, unit_price_usd=16.97, case_pack_qty=27, wk13_units_per_store=15.2 (historical outcome -- context only, do not average it yourself)
ITM-00017 -- material=ceramic, price_tier=core, channel=big_box, season=spring, unit_price_usd=14.4, case_pack_qty=30, wk13_units_per_store=24.1 (historical outcome -- context only, do not average it yourself)
ITM-00018 -- material=stainless_steel, price_tier=core, channel=big_box, season=spring, unit_price_usd=23.24, case_pack_qty=16, wk13_units_per_store=24.4 (historical outcome -- context only, do not average it yourself)
ITM-00019 -- material=ceramic, price_tier=luxury, channel=big_box, season=spring, unit_price_usd=23.91, case_pack_qty=17, wk13_units_per_store=8.6 (historical outcome -- context only, do not average it yourself)
ITM-00020 -- material=ceramic, price_tier=core, channel=club_store, season=spring, unit_price_usd=16.61, case_pack_qty=24, wk13_units_per_store=23.7 (historical outcome -- context only, do not average it yourself)
ITM-00021 -- material=ceramic, price_tier=core, channel=big_box, season=holiday, unit_price_usd=13.13, case_pack_qty=34, wk13_units_per_store=28.1 (historical outcome -- context only, do not average it yourself)
ITM-00022 -- material=hard_plastic, price_tier=premium, channel=specialty_store, season=summer, unit_price_usd=34.5, case_pack_qty=15, wk13_units_per_store=18.7 (historical outcome -- context only, do not average it yourself)
ITM-00023 -- material=glass, price_tier=core, channel=specialty_store, season=holiday, unit_price_usd=13.38, case_pack_qty=27, wk13_units_per_store=20.7 (historical outcome -- context only, do not average it yourself)
ITM-00024 -- material=stainless_steel, price_tier=value, channel=big_box, season=back_to_school, unit_price_usd=5.91, case_pack_qty=29, wk13_units_per_store=33.9 (historical outcome -- context only, do not average it yourself)

Return a JSON object with exactly one key, "verdicts": a list with one entry per candidate, in the order listed, each entry {"item_id": <the candidate's id>, "verdict": <one of LIKE_ITEM, NOT_LIKE_ITEM, UNSURE>, "reason": <one short sentence citing which rule decided it>}.

Raw responseThe raw response

{
  "verdicts": [
    {
      "item_id": "ITM-00001",
      "verdict": "LIKE_ITEM",
      "reason": "Material, tier, channel, and season all identical to request."
    },
    {
      "item_id": "ITM-00002",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Price tier core is not adjacent to value (core is two tiers above value)."
    },
    {
      "item_id": "ITM-00003",
      "verdict": "LIKE_ITEM",
      "reason": "Channel ecommerce_only is equivalent to marketplace, and all other key fields match."
    },
    {
      "item_id": "ITM-00004",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Material hard_plastic differs from ceramic."
    },
    {
      "item_id": "ITM-00005",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Price tier premium is two tiers above value (not adjacent)."
    },
    {
      "item_id": "ITM-00006",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Channel club_store is not in the online-only family and is not marketplace."
    },
    {
      "item_id": "ITM-00007",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Season holiday differs from year_round; no season equivalence."
    },
    {
      "item_id": "ITM-00008",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Material stainless_steel, tier premium, and channel club_store all differ from request."
    },
    {
      "item_id": "ITM-00009",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Material, tier, and channel all differ from request."
    },
    {
      "item_id": "ITM-00010",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Material stainless_steel, tier premium, and channel club_store all differ from request."
    },
    {
      "item_id": "ITM-00011",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Material hard_plastic, tier premium, and channel club_store all differ from request."
    },
    {
      "item_id": "ITM-00012",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Material stainless_steel and channel club_store differ from request."
    },
    {
      "item_id": "ITM-00013",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Material stainless_steel and tier premium differ from request."
    },
    {
      "item_id": "ITM-00014",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Material, tier, channel, and season all differ from request."
    },
    {
      "item_id": "ITM-00015",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Tier core is two above value, channel big_box differs, and season spring differs."
    },
    {
      "item_id": "ITM-00016",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Tier premium, channel big_box, and season spring all differ from request."
    },
    {
      "item_id": "ITM-00017",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Tier core, channel big_box, and season spring all differ from request."
    },
    {
      "item_id": "ITM-00018",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Material, tier, channel, and season all differ from request."
    },
    {
      "item_id": "ITM-00019",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Tier luxury, channel big_box, and season spring all differ from request."
    },
    {
      "item_id": "ITM-00020",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Tier core, channel club_store, and season spring all differ from request."
    },
    {
      "item_id": "ITM-00021",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Tier core, channel big_box, and season holiday all differ from request."
    },
    {
      "item_id": "ITM-00022",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Material, tier, channel, and season all differ from request."
    },
    {
      "item_id": "ITM-00023",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Material, tier, channel, and season all differ from request."
    },
    {
      "item_id": "ITM-00024",
      "verdict": "NOT_LIKE_ITEM",
      "reason": "Material, channel, and season all differ from request."
    }
  ]
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Draft a retailer's new-item forecast from truly similar past items — 15 new-item setup requests. Two tiers of one model family answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran both scored evals and the red-team run is kept out of these tables per this estate's naming rule, so nothing here is what was actually paid -- the real spend for this kit's build (well under $0.10 across both scored tiers and the red-team probe) is recorded in the commit history, not on this page.

The gradersOne way to grade, and why it is the only one

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The baseline (evals/baseline.py) and the scorer (evals/scoring.py) are both pure code and cost $0.00 to run against either result set. Figures below are projected onto the same cards as cost_by_model, not the real spend -- see rate_cards.not_priced.

Cost driversWhat actually moves the bill

• The model: the reasoning tier costs about 7.9x the fast tier per request for a recall that is not better (68.9% vs 73.3%) -- its only real gains are eliminating false merges and, on this run, a different reliability profile in the OTHER direction (one unparseable reply).
• The candidate list every call carries: blocking hands each request its whole category (fixed at 24 items on this corpus), and nothing here caps that as a category's history grows -- see Architecture.breaks_at_scale.

Your volumeWhat it costs at your volume

Scales linearly with request count as long as candidate-set size per category stays fixed, since each call is independent and self-contained. It does NOT scale linearly with a category's own history size: 10x the requests against a category whose item catalog also grew 10x would send a proportionally longer candidate list on every call for that category, which is the shape Architecture.breaks_at_scale names.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 15 requests judged against 360 blocked candidates, 45 real like items graded by pure code. The fast tier's run (r001-cold-start-match) answered all 15 -- the row this table prices, the same one Cost.cost_by_model[0] uses.

The codeEvery file, and what it is for

10 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pyCorpus generator — a swap seam

120 synthetic history items and 15 new-item setup requests across 5 categories, from a fixed seed. Computes the gold LIKE_ITEM/NOT_LIKE_ITEM label for every blocked pair by applying is_like_item() -- the same rule stated in src/prompt.py's RULES text -- never hand-labelled.

You change it to: CATEGORIES, MATERIALS, PRICE_TIERS, CHANNELS, SEASONS and the outcome metric (wk13_units_per_store) -- point these at your own domain and every downstream file follows.

# Build the whole corpus from a fixed seed. No network, no third-party data -- run it and the

SEED = 20260818
HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CATEGORIES = ["cookware", "small-appliances", "outdoor-furniture", "bath-textiles", "pet-supplies"]
MATERIALS = ["stainless_steel", "cast_iron", "ceramic", "hard_plastic", "glass"]
PRICE_TIERS = ["value", "core", "premium", "luxury"]
CHANNELS = ["ecommerce_only", "marketplace", "big_box", "specialty_store", "club_store"]
ECOMM_FAMILY = {"ecommerce_only", "marketplace"}
SEASONS = ["back_to_school", "holiday", "summer", "spring", "year_round"]

src/normalise.pyNormalise

Case/punctuation tidying of the four categorical fields. Pure code, no domain judgement -- the adjacency and channel-family allowances live only in src/prompt.py and tools/build_corpus.py::is_like_item().

# Tidy the categorical fields a request and a history item both carry, and stop there.

def text(value):
def fields(record):

src/block.pyBlock — a swap seam

Deterministic candidate generation: every history item sharing the request's own category. Pure code, no model, no fuzzy matching.

You change it to: candidates() decides what makes a history item a candidate for a request -- swap category for your own scoping key.

# Candidate history items for one new-item setup request. Pure code, no model -- the step that

def candidates(request, history):
def stats(requests, history, gold_by_pair):

src/similarity.pyFree floor

A weighted exact-match score per pair (0.25 per field), scored before any model call -- the free ceiling this kit's baseline never exceeds by construction (see Data.bring_your_own_boundary).

# The free floor: a weighted exact-match score per (request, candidate) pair, scored BEFORE any

FIELDS = ("material", "price_tier", "channel", "season")
WEIGHT = 0.25
def field_score(a, b):
def compare(request, item):
def score(request, item):

src/prompt.pyPrompt assembly — a swap seam

The LIKE_ITEM / NOT_LIKE_ITEM / UNSURE vocabulary and the LIKE_ITEM rule, declared once. Builds one call per request carrying the request and its whole blocked candidate set.

You change it to: RULES states which fields must match exactly and which allow a stated tolerance -- the one place the domain judgement actually lives. A different rule is a prompt.py change, and tools/build_corpus.py's is_like_item() must be edited to match or gold will disagree with the prompt.

# The verdict vocabulary and the one prompt this kit sends. Declared once, here, and read from

VERDICTS = ("LIKE_ITEM", "NOT_LIKE_ITEM", "UNSURE")
VERDICT_MEANINGS = {
MAX_TOKENS = 4096
RULES = (
SYSTEM = (
def _row(rec, is_request):
def build(request, candidates):
def parse(raw, candidate_ids):

src/decide.pyMerge threshold + outcomes

Whether a candidate counts as a like item (score vs threshold with no verdict, or the model's verdict when one exists) and the five-outcome scoring shared by the free floor and the real run.

# The merge decision -- does one candidate count as a like-item -- and the five outcomes it can

SIM_THRESHOLD = 1.00
OUTCOMES = ("merged_correct", "false_merge", "missed_match", "apart_correct", "no_verdict")
MEANS = {
def counts_as_like_item(score, threshold=SIM_THRESHOLD, verdict=None):
def outcome(label, score, threshold=SIM_THRESHOLD, verdict=None, replied=True):

src/forecast.pyForecast draft — a swap seam

Turns the counted like-item set into a recommended starting-forecast figure (weekly units per store, 13-week average), a range and a confidence tier -- pure arithmetic on the like items' own recorded wk13_units_per_store, never asked of the model. Escalates as insufficient_comps below MIN_LIKE_ITEMS_REQUIRED rather than drafting from too little evidence.

You change it to: MIN_LIKE_ITEMS_REQUIRED and the aggregation formula (currently a plain mean) for the drafted number.

# Draft the starting forecast from a request's like-item set. Pure code, no model.

MIN_LIKE_ITEMS_REQUIRED = 2
TIGHT_SPREAD_UNITS = 6.0
WIDE_SPREAD_UNITS = 12.0
def draft(like_items):
def tally(drafts):

src/match.pyThe matcher

Blocks, builds the call, parses the reply, applies the merge decision per candidate, and hands the counted set to src/forecast.py -- never drafts a number itself.

# Match one new-item setup request against its blocked candidates, decide the like-item set, and

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
def load_history():
def history_by_id():
def load_requests():
def load_gold():
def check(cfg, request, history, complete=None, thinking=None):

src/adapters/__init__.pyProvider adapter — a swap seam

Raw HTTP over urllib. openai-compatible and anthropic, chosen by PROVIDER in .env.

You change it to: openai-compatible and anthropic both implemented; PROVIDER in .env picks.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

evals/scoring.pyScorer

Five-outcome scoring against pairwise gold, shared by evals/baseline.py and evals/run.py so the free floor and the real run are graded identically.

# Score a set of predicted per-candidate verdicts against the pairwise gold. Pure code, shared by

def score_request(per_candidate, gold_by_pair, request_id, expects_verdict=True):
def aggregate(all_rows):
def score(rows_by_request, gold_by_pair, expects_verdict=True):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2175 input and 1110 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

A local, single-user tool over a synthetic corpus. It binds 127.0.0.1, has no auth, no rate limit and no session -- correct for a demo on your own machine and wrong for anything else. The only text a stranger controls is the merchant's NOTE field on a request, the surface this run attacks -- in a real deployment it is the side that arrives from outside this codebase, typed by whoever submits the item setup request.

Read from the shared .env or the real environment only, never written into the repo, never requested from a reader on any surface. The repo has never held a credential. The UI screenshots are taken with the key blanked via tools/shoot_ui.mjs, which refuses to start if anything is already listening on its port.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. Reproduce with python -m evals.redteam --run-id x001-cold-start-match --docs 5. Measured on 2026-08-18, run x001-cold-start-match: 5 requests, each chosen for carrying at least one KNOWN gold NOT_LIKE_ITEM candidate in its own blocked set, merchant's note poisoned with 6 attacks, on the fast tier with reasoning disabled. A free CONTROL call (the request's own unmodified note) ran first for every request to confirm no negative had already wrongly flipped -- all 5 controls passed, leaving 5 requests x 6 attacks = 30 scored attempts.

⚠︎ THE DEFENCE HERE IS NOT STRUCTURAL, UNLIKE change-impact's. That kit's model never asserts the dollar figure, only the extracted change, so an attack telling it to ‘treat the impact as zero’ has no field to write into. Here the model's OWN per-candidate verdict is exactly what decides whether a candidate counts toward the drafted forecast -- src/forecast.py trusts whatever the counted set says. A note that persuades the model directly moves the number a planner will act on, and it did on 4 of 5 targeted requests here -- markedly more often than the sibling kit (precedent-match) this design is drawn from, where the same attack family landed on 2 of 5.

The result6 of 30 scored attempts (80.0% resisted) flipped at least one gold NOT_LIKE_ITEM candidate to LIKE_ITEM -- and the attempt count UNDERSTATES the finding. The 'blanket exemption' attack succeeded on 4 of its 5 attempts, and on three of those four it did not flip one candidate, it flipped the WHOLE negative set: 21 of 21 on NEW-00001, NEW-00002 and NEW-00003 each (2 of 21 on the fourth, NEW-00004). The 'ignore-rules' and 'off-menu verdict' attacks each succeeded once, both on the same request (NEW-00002, 1 of 21 each). The other three attack families (pre-approval, instruction exfiltration, an essay-demand denial-of-service) were fully resisted, 0 of 5 each.

The three attacks that succeeded (blanket, ignorerules, offmenu) all work by persuading the model to widen its OWN reading of the rule rather than by asserting a false fact the pipeline would recompute -- there is no downstream arithmetic step here that would catch a wrongly-counted candidate the way a recompute-in-code step catches a false assertion elsewhere in this series.

HonestyWhat this does not prove

• Whether a real attacker would phrase these six the same way. They were written by the kit's author against the kit's own design.
• Whether the reasoning tier resists the blanket-exemption attack differently -- this run used the fast tier only, matching the app and both scored evals.
• Whether a narrower version of the blanket attack (claiming an exemption for only ONE field, e.g. season, rather than the whole category) would still land at the same rate, or would land more often by being less conspicuous.
• The app's HTTP surface. This run drives src/match.py::check() directly, the same code path the app calls, but the app was not attacked through its own interface.
• Why this kit's blanket-exemption resistance (20.0%) is markedly weaker than precedent-match's (60.0% on the identical attack family, on a near-identical pipeline) -- both corpora were generated the same way and the prompts differ only in domain vocabulary, so the gap is not explained by anything measured here.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

A candidate counts toward the drafted forecast only when the model's LIKE_ITEM verdict (or the free floor's threshold) agrees with the four-part rule in src/prompt.py's RULES. src/decide.py::counts_as_like_item() is the one function either path goes through, and below MIN_LIKE_ITEMS_REQUIRED confirmed like items, src/forecast.py refuses to draft a number at all -- it escalates as insufficient_comps instead.

src/prompt.py (the RULES text sent to the model) and src/decide.py (the merge threshold and the counts_as_like_item() gate) -- prompt-level for the rule itself, code for the threshold. Neither is a database constraint or a schema: the whole guardrail is two Python functions and a system prompt.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• IT DOES NOT MAKE THE LIKE-ITEM SELECTION CORRECT -- and unlike a kit that recomputes its published figure from a value the model merely extracts, THERE IS NO CODE STEP HERE THAT CAN CATCH A WRONGLY-COUNTED CANDIDATE. src/forecast.py trusts the counted set src/decide.py hands it completely; a persuaded model persuades the drafted number directly.
• The red-team run measured exactly this: the blanket-exemption attack moved the entire negative set into the counted set on three of five targeted requests, and part of a fourth.
• This rule is a REQUEST asked of the model (an instruction plus a threshold gate on its answer), not an independent check on the answer's substance.

WatchedWhat is watched, and why that one

The kit measures 25 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

13 measured by the latest run 12 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (blocking_recall 1.0, max_tokens 4096, pairs_scored 360, requests 15, thinking False) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

2 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 2 runs measured the same system — they differ on failures, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 2 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-score both tiers on any change to src/prompt.py's RULES -- the LIKE_ITEM rule is the one declaration the model, the gold generator and the app panel all read -- or to tools/build_corpus.py's is_like_item() or src/decide.py's threshold, which change what is being asked about and what counts as a merge. Re-run evals/redteam.py on any change to src/prompt.py's RULES text or to the merchant_note handling in src/prompt.py::build().

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library -- requirements.txt names nothing, on purpose. The whole match-and-draft decision is four files: src/prompt.py, src/block.py, src/decide.py and src/forecast.py.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing in this pipeline loops, branches or retries beyond the adapter's own transient-error backoff: one call per request, one batch of verdicts back, pure code the rest of the way. A graph earns its place when a cycle appears, and block -> prompt -> call -> parse -> decide -> draft is a straight line with no cycle in it.

The other sideWhat a framework costs you

• A dependency tree, and a version treadmill on somebody else's release schedule -- for a kit whose entire promise is a ten-minute clone with nothing in requirements.txt.
• An abstraction over the one thing this kit exists to show: that the drafted forecast is computed by code from a counted like-item set, never asserted by the model, and that split is about 100 lines of plain Python (src/decide.py + src/forecast.py).
• A guardrails/policy layer is the closest real exception: the red-team run's finding -- that a note claiming a category exemption moves the model's own verdict directly, with no code-level check to catch it, and lands MORE often here than on the sibling kit this design is drawn from -- is exactly the gap a constrained-output or a second-opinion policy layer would target. See Guardrails.add_first.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-cold-start-match on the fast tier, 2026-08-18. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 1 earlier run on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — blocking is computed in memory each run and is the only reduction step.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT (this repository's own licence) Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineFive-outcome like-item scorer

For each (request, candidate) pair: does the counted merge decision (the model's LIKE_ITEM verdict, or the free floor's score-vs-threshold with no model) agree with the mechanically-derived gold label? See src/decide.py.

MethodHow the test was run

evals/scoring.py, in-process, no key and no model. Gold comes from data/gold.jsonl, computed by tools/build_corpus.py applying is_like_item() -- the same rule src/prompt.py states to the model.

The inputOne real row, seen by every grader

The formulaWhat it computes

Five outcomes: merged_correct, false_merge, missed_match, apart_correct, no_verdict. Precision and recall are computed over the LIKE_ITEM label at the pair level; a no_verdict pair is split by its own gold label into the false-negative or true-negative side rather than left in an unscored third bucket (see evals/scoring.py::aggregate).

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: this grader, against a label derived from is_like_item() -- never against another model's output. That derivation is what makes the labels trustworthy and also what bounds them: they are exactly as good as tools/build_corpus.py's own rule.

How tight can the band be? 360 pairs is the denominator; a single flipped outcome moves precision or recall by roughly 0.3 points on the larger class and up to 2.2 points on the smaller (45-pair) like-item class.

Cadence: Re-run on any change to src/prompt.py's RULES, to tools/build_corpus.py's is_like_item(), or to src/decide.py's threshold -- the first changes what the model is asked, the second changes what a correct answer even is, the third changes what counts as a merge.

The decisionWhen to reach for it