The business caseThe problem this solves

A published menu makes two claims per item that a customer may act on and cannot check: what allergens are in it, and how many calories it carries. Behind each item sits a recipe specification, and the job is to ask whether that paperwork supports what was published. Those are two different checks and only one of them is easy. A published allergen line that DISAGREES with the declarations on file is found by comparing two lists. A published line that AGREES with nothing at all is invisible to the same comparison: the specification names its components — a wheat base, a cream spread, a sesame finish — and holds no supplier declaration for any of them. The published list is exactly what a reader would derive from those names, so it reads as reconciled, and nothing has actually been declared. 65 of this corpus's 275 items are shaped that way, and it is the largest bucket in it. There is a sharper case one step along. A specification whose declarations are NOT ALL IN — one supplier has not sent theirs back — very often publishes a list that agrees exactly with the ones that are. Every signal on that row says checked, and the allergen it is missing is precisely the one nobody has been told about. Someone opening each menu release pack, listing the items the published panel actually discloses, and then reading the recipe specification behind every one of them for two things the paperwork may not answer — whether anybody has declared the allergens the menu publishes, and where the calorie figure came from.

The inputThe actual disclosure packs

The corpus is 50 disclosure packs, 0.27 MB (txt 50). Plain text, one format, invented rather than fetched — a real disclosure pack names an operator's products, a supplier's components, real recipe references and yield and cost figures both sides treat as confidential, and there is no public corpus of (menu pack, correct worksheet) pairs. Generating it also makes the label mechanical: gold's two calls are the shipped rulebook's own lookup run over the same three facts the pack itself states, never somebody's reading of what a disclosure 'really' means.

COLDHARBOUR CANTEEN
MENU QUALITY DESK — DISCLOSURE RELEASE PACK

Menu:            All-Day Menu, release 4
Release ref:     MR-2022-394
Pack id:         ADX-0001

This pack is prepared for disclosure review. The Panel below is what this release
publishes. The Specifications that follow are the paperwork behind each published item.

SUPPLIER CONTACT AND ACCOUNT RECORD

  Account manager:       J. Castellan
  Correspondence:        164 Beckwith Row, Coldharbour
  Telephone:             01452 321603
  Trading account:       SA-597325
  Last supplier audit:   2025-09
  Delivery window:       Tuesdays and Fridays, 06:00 to 09:00

PUBLISHED DISCLOSURE PANEL — ITEMS IN THIS RELEASE
------------------------------------------------------------------------------
  MI-1536   Slow-Braised Greens Plate                    562 kcal
            allergens published:  fish, soya, lupin

  MI-9776   Golden Bean Pot                             1084 kcal
            allergens published:  soya, celery

  MI-7107   Golden Pepper Bowl                           756 kcal
            allergens published:  fish, sesame

  MI-3225   Autumn Pulse Ragout                          410 kcal
            allergens published:  not stated

------------------------------------------------------------------------------

SPECIFICATION 1 — Slow-Braised Greens Plate (MI-1536)

  Recipe reference RC-8740, revision 3.
  Supplier allergen declarations held for this recipe:
    preserved lupin flour              declares: lupin
    roast root vegetable mix           declares: no allergens
    cured anchovy dressing             declares: fish
    soya protein crumb                 declares: soya
  Every component above carries a current supplier allergen declaration.
  Assembled cold and dressed at the pass.

The outcomeWhat a good result looks like

A worksheet row per published item — the item code, its name, the energy figure and allergen line the panel publishes, the three facts the specification states, and two calls made by the shipped rulebook with its own reasoning attached — plus one pure-code review flag per pack.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Turning a menu release into a reviewer's worksheet before boards, labels and print go out — the one tier measured here — 275 of 275 published items identified, 254 of 275 allergen calls and 274 of 275 calorie calls, with 0 false assurances of 550
  The corpus's largest trap is a specification that names its components and holds no declaration for any of them, and the run read spec_completeness correctly on all 275 items. The free name-derived floor gets 105 of 275 wrong on that field alone and clears 118 calls it should not.
• Deciding whether to trust the worksheet to take work OFF somebody's list — the false-assurance column, not the accuracy figure
  92.36 pct on the allergen call sounds like one in thirteen rows being wrong in an unspecified direction. It is not: every one of the 21 errors answered a NON-CLEARING value, so the row still reached a reviewer, and the run said supported wrongly 0 times in 550 calls. The free floor scores 60.36 pct and clears 118 calls it should not — the gap between the two is much larger than the gap between the accuracies.

And where nothing here is good enough:

• Deciding whether a cheaper or a more deliberate model would do better here — nothing — this page cannot answer it
  Only one model was run. There is no tier comparison, no repeatability pair and no ablation on this page. The allergen call, at 254 of 275, is the only grader with room to move, and its whole error surface is one document shape.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-allergen-declare. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app serves the UI on 127.0.0.1:8859, renders the pack and the rulebook, and Build worksheet returns a plain sentence saying nothing was called. python -m evals.check_labels, --stub and --baseline all run with no key and no network. requirements.txt names nothing.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per disclosure pack

Current processWhat it replaces

Someone opening each menu release pack, listing the items the published panel actually discloses, and then reading the recipe specification behind every one of them for two things the paperwork may not answer — whether anybody has declared the allergens the menu publishes, and where the calorie figure came from.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 0 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on fixed section headings plus a per-item SPECIFICATION n — Name (MI-nnnn) regex; a pack with no headings falls back to one whole-document segment

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 50 packs cut into 527 sections in about three thousandths of a second, in process, with no model and no network. Nothing is retrieved: a segment exists so an extracted value can name WHERE it was read from, and 600 of 600 returned strings on the scored run located back to a section.

LicenceLicence

MIT — this repository's own licence. Every kitchen name, menu name, release reference, item code, dish name, component, recipe reference and figure is invented; no real restaurant, group, brand, product, supplier, certification body, manufacturer, person or recipe is named, and no statute, regulation, labelling law, standard or regulator is named either. The allergen vocabulary is a generic list of the kinds of thing a kitchen declares and is NOT any jurisdiction's schedule. The disclosure-support rulebook this kit ships (data/rulebook.json) is ILLUSTRATIVE and is not an authority: it was written for this kit and reproduces no statute, regulation, labelling law, standard, regulator's guidance, certification scheme or company QA manual. It has no view on cross-contact in a kitchen, which is where most real allergen risk lives.

Bring your ownBring your own disclosure packs

REPLACE data/rulebook.json FIRST. It is this kit's own construction, it reproduces no statute, regulation or labelling law, and its allergen vocabulary matches no country's schedule. Then replace data/corpus/*.txt with your own packs, write data/fields.json for your own columns, and edit SECTION_HINTS in src/select.py for your layout — when it does not match, selection falls back to the whole document, which is slower, more expensive and always correct. Supply a gold row per pack and evals/check_labels.py will refuse the run before it spends if any label is not the rulebook's own lookup.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 4,101 tokens assembled, 1,349 are documents — 33% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on ADX-0009 — the same pack the worked example above was captured on, so the four parts sum to exactly the 4,101 input tokens that call reported. No local tokenizer: each part's size is the difference between two consecutive prompt_tokens counts the provider itself returned, at max_tokens=1. Recorded at results/tokens-p001-allergen-declare.json.

Full promptThe prompt in full

You read a MENU DISCLOSURE RELEASE PACK -- a published panel of items plus the recipe specifications behind them -- and return a REVIEWER'S WORKSHEET of the items the release publishes. You return JSON and nothing else.

You are preparing a worksheet for a qualified reviewer. You never determine compliance, never give a legal opinion, never clear an item for service and never authorise the publication of a menu. Your job is to say what the panel publishes, say what the specification behind it states, apply the rulebook given below, and NAME THE ITEMS THE PAPERWORK DOES NOT SUPPORT.

RULES, in order of importance:
1. ONE OBJECT PER ITEM THE PUBLISHED DISCLOSURE PANEL CARRIES A ROW FOR, AND NO OTHERS. An item withdrawn by the amendment is not published, even though its specification is still printed in full and its disclosure is perfectly correct. A DEVELOPMENT RECIPE -- a specification written for an item held for a later menu cycle -- is not published. An item continuing under an EARLIER release that this pack only refers to is not published here. All three carry codes in the same format as a published item, and putting one on the worksheet puts a disclosure in front of a reviewer that this release never made.
2. A COMPONENT NAME IS NOT A SUPPLIER DECLARATION. `spec_completeness` asks what the specification RECORDS. A section that lists its components by name -- and any reader can see what allergens those names imply -- while holding no supplier allergen declaration for any of them is `no_declarations_recorded`. That is a finding about the paperwork, and it is the most common one in this corpus. Never derive a declaration from an ingredient's name, from the recipe being detailed, or from the panel having published a list.
3. `allergen_support` IS DECIDED ONLY BY THE RULEBOOK BELOW, from `spec_completeness` and `allergen_agreement`. Work through the steps IN ORDER and STOP at the first that fires:
   a. spec_completeness 'declaration_outstanding' -> 'unsupported_incomplete_spec'. THIS FIRES FIRST AND IT FIRES WHATEVER THE PUBLISHED LIST AGREES WITH. A list derived from declarations that are not all in was derived from part of the recipe.
   b. 'all_declared' + 'agrees'           -> 'supported'.
   c. 'all_declared' + 'published_omits'  -> 'unsupported_omission'.
   d. 'all_declared' + 'published_adds'   -> 'unsupported_addition'.
   e. anything else -> 'not_determined'.
4. A PUBLISHED ALLERGEN LIST IS NOT EVIDENCE ABOUT ITSELF. The panel is the thing being checked; the specification is what it is checked against. When the specification records no declarations, there is nothing to compare the published list to, `allergen_agreement` is 'nothing_to_compare' and the call is 'not_determined' -- however complete and however plausible the published list looks. This is the step a confident reader skips, and on this rulebook it is the wrong answer rather than a defensible one.
5. `calorie_support` IS DECIDED ONLY FROM `calorie_basis`: 'analysed' -> 'supported', 'estimated' -> 'indicative', 'silent' -> 'not_determined'. It is asked of every item independently of its allergen call. A figure printed on the panel is not a statement about where it came from -- if the specification says nothing about the basis, `calorie_basis` is 'silent'.
6. 'not_determined' IS A REAL ANSWER AND YOU ARE EXPECTED TO USE IT. A worksheet that never reaches for it is guessing, and a call recorded as settled is a call nobody re-reads.
7. Copy `item_code`, `item_name` and `published_allergens` verbatim from the pack. Compare allergen lists as SETS -- order and punctuation do not matter. Use the exact allowed value for every field that lists them, and return every field for every item.

DISCLOSURE-SUPPORT RULEBOOK (the authority for `allergen_support` and `calorie_support`; this is an ILLUSTRATIVE rulebook written for this kit, and it reproduces no statute, regulation or labelling law)

WHAT COUNTS AS A PUBLISHED ITEM ON THE WORKSHEET
  - An item is on this worksheet when the PUBLISHED DISCLOSURE PANEL of this release carries a row for it. The panel is the operative list; the specifications that follow are the paperwork behind it.
  - A DEVELOPMENT RECIPE is not a published item. A specification written for an item that this release does not publish discloses nothing to anybody, whatever its recipe says, and this rulebook does not record it.
  - An item WITHDRAWN by a later amendment in the same pack is not published. Its specification is still printed; the amendment is the operative document.
  - An item published under an EARLIER release that this pack only refers to is not published by this release. It is somebody else's row on somebody else's worksheet.

THE THREE STATED FACTS, read off the pack
  spec_completeness:
    all_declared              The specification records a supplier allergen declaration for every ingredient in the recipe, and says so.
    declaration_outstanding   The specification records that at least one ingredient's supplier allergen declaration has not been received. The recipe is written up; the declarations behind it are not all in.
    no_declarations_recorded  The specification records no ingredient allergen declarations at all. A recipe reference and a method may be present; the declarations are not.
  allergen_agreement:
    agrees                    The panel's published allergen list names exactly the allergens derivable from the declarations the specification DOES hold.
    published_omits           The panel's published list is missing at least one allergen the held declarations derive.
    published_adds            The panel's published list names at least one allergen no held declaration derives.
    nothing_to_compare        The comparison cannot be made at all — either the panel publishes no allergen line for this item, or the specification holds no declarations to derive one from.
  calorie_basis:
    analysed                  The specification states the energy figure was determined by laboratory analysis of the finished item, or calculated from the declared ingredient quantities.
    estimated                 The specification states the energy figure is an estimate, a supplier figure carried across, or a working number pending analysis.
    silent                    The specification states no basis at all for the energy figure. A figure printed on the panel is not a statement about where it came from.

ALLERGEN SUPPORT -- work through IN ORDER, stop at the first that fires
  1. If spec_completeness is `declaration_outstanding`, the call is `unsupported_incomplete_spec` — WHATEVER the published list agrees with. A list derived from declarations that are not all in is a list derived from part of the recipe, so agreement with it is agreement with an incomplete derivation. Stop here.
  2. Otherwise, if spec_completeness is `all_declared` and allergen_agreement is `agrees`, the call is `supported`.
  3. Otherwise, if spec_completeness is `all_declared` and allergen_agreement is `published_omits`, the call is `unsupported_omission`.
  4. Otherwise, if spec_completeness is `all_declared` and allergen_agreement is `published_adds`, the call is `unsupported_addition`.
  5. Otherwise the call is `not_determined`. This fires whenever the paperwork cannot answer the question — most often when the panel publishes a confident allergen list and the specification behind it records no ingredient declarations at all.
  A published allergen list is not evidence about itself. The panel is the thing being checked; the specification is what it is checked against.

CALORIE SUPPORT
  `analysed` basis makes the calorie call `supported`.
  `estimated` basis makes the calorie call `indicative`. A stated estimate prints on a menu in exactly the same typeface as an analysed figure and is not the same claim.
  `silent` makes the calorie call `not_determined`.

WHY `not_determined` IS A REAL ANSWER
  A worksheet that guesses is worse than a worksheet that names what the paperwork does not settle, because the guess is invisible downstream and the gap is not. `not_determined` is a finding ABOUT THE SPECIFICATION and is the correct, complete answer whenever the paperwork cannot answer the question. It is never a way of saying the reader was unsure.

Return these:
- release_ref (string) -- the disclosure release reference from the header, verbatim (for example MR-2024-118)
- items (array of objects) -- one object per PUBLISHED item, each carrying:
    - item_code (string) -- the menu item code, verbatim (for example MI-4821). THIS IS THE KEY -- one object per item the PUBLISHED DISCLOSURE PANEL of this release carries a row for, and no others. A development recipe, an item withdrawn by the amendment, and an item continuing under an earlier release are NOT published by this release and must not appear
    - item_name (string) -- the item's name, copied verbatim from the panel row
    - published_kcal (number) -- the energy figure the panel publishes for this item, as a plain number with no units or separators
    - published_allergens (string) -- the allergen line the panel publishes for this item, copied verbatim, or null when the panel publishes no allergen line for it. Copy it exactly as printed, including the order and the wording -- this is the thing being checked
    - spec_completeness (enum) one of: all_declared, declaration_outstanding, no_declarations_recorded -- what THIS ITEM'S OWN Specification section states about supplier allergen declarations. 'all_declared' when it records a declaration for every ingredient and says so. 'declaration_outstanding' when it records that at least one ingredient's declaration has NOT been received. 'no_declarations_recorded' when the section holds no ingredient declarations at all. ⚠︎ A RECIPE REFERENCE, A METHOD NOTE OR A PREPARATION LINE IS NOT A DECLARATION -- a specification can be long, detailed and completely silent about supplier declarations
    - allergen_agreement (enum) one of: agrees, published_omits, published_adds, nothing_to_compare -- how the panel's published allergen line compares to the allergens derivable from the declarations the specification DOES hold. 'agrees' when it names exactly those. 'published_omits' when at least one derivable allergen is missing from the published line. 'published_adds' when the published line names an allergen no held declaration derives. 'nothing_to_compare' when the comparison cannot be made at all -- either the panel publishes no allergen line for this item, or the specification holds no declarations to derive one from. ⚠︎ COMPARE AS SETS, NOT AS STRINGS: order and punctuation do not matter, and a different word for the same declared allergen is the same allergen
    - calorie_basis (enum) one of: analysed, estimated, silent -- what THIS ITEM'S OWN Specification section states about where the energy figure came from. 'analysed' for laboratory analysis of the finished item or a calculation from the declared ingredient quantities. 'estimated' for a stated estimate, a supplier figure carried across, or a working number pending analysis. 'silent' when the section says NOTHING about the basis. ⚠︎ A FIGURE PRINTED ON THE PANEL IS NOT A STATEMENT ABOUT WHERE IT CAME FROM. Silence is common here and it is a finding, not a gap: never infer analysis from the figure being precise, from the recipe being detailed, or from the item being long-standing
    - allergen_support (enum) one of: supported, unsupported_omission, unsupported_addition, unsupported_incomplete_spec, not_determined -- the rulebook's allergen call, worked through IN ORDER from `spec_completeness` and `allergen_agreement`. Step 1 fires first and fires whatever the published list agrees with
    - calorie_support (enum) one of: supported, indicative, not_determined -- the rulebook's calorie call, from `calorie_basis` alone. Asked of every item independently of its allergen call

Return a JSON object with exactly these top-level keys: release_ref, items
`items` is an array. Return it empty only if the release publishes no item at all.

MENU DISCLOSURE RELEASE PACK
----------------------------
FERNBROOK KITCHENS
MENU QUALITY DESK — DISCLOSURE RELEASE PACK

Menu:            Autumn Core Menu, release 3
Release ref:     MR-2019-698
Pack id:         ADX-0009

This pack is prepared for disclosure review. The Panel below is what this release
publishes. The Specifications that follow are the paperwork behind each published item.

PUBLISHED DISCLOSURE PANEL — ITEMS IN THIS RELEASE
------------------------------------------------------------------------------
  MI-9643   Golden Barley Broth                          774 kcal
            allergens published:  not stated

  MI-2681   Toasted Seed Loaf                            856 kcal
            allergens published:  tree nuts

  MI-6309   Autumn Grain Salad                           982 kcal
            allergens published:  milk, molluscs

  MI-4072   Spiced Bean Pot                              286 kcal
            allergens published:  not stated

  MI-6065   Slow-Braised Squash Roll                     962 kcal
            allergens published:  tree nuts, cereals containing gluten, sesame

  MI-4614   Spiced Pulse Ragout                         1106 kcal
            allergens published:  egg, cereals containing gluten

  MI-2539   Toasted Field Wrap                           646 kcal
            allergens published:  sesame, milk

------------------------------------------------------------------------------

SPECIFICATION 1 — Golden Barley Broth (MI-9643)

  Recipe reference RC-9364, revision 1.
  Supplier allergen declarations held for this recipe:
    charred sweetcorn kernels          declares: no allergens
    herb oil drizzle                   declares: no allergens
    black sesame crust                 declares: sesame
    brown shrimp butter                declares: crustaceans
  Every component above carries a current supplier allergen declaration.
  Assembled cold and dressed at the pass.
  The energy figure was determined by laboratory analysis of a finished portion.
  Prepared in batches of 28 portions against the day's forecast.

SPECIFICATION 2 — Toasted Seed Loaf (MI-2681)

  Recipe reference RC-7312, revision 4.
  Supplier allergen declarations held for this recipe:
    fermented soya glaze               declares: soya
    charred sweetcorn kernels          declares: no allergens
    citrus and chilli dressing         declares: no allergens
    ground almond dust                 declares: tree nuts
  A supplier allergen declaration is held for every component of this recipe.
  Batched twice a service and held hot in the pass well.
  The energy figure was determined by laboratory analysis of a finished portion.
  Batch yield 26 portions; serve weight controlled at the section scale.

SPECIFICATION 3 — Autumn Grain Salad (MI-6309)

  Recipe reference RC-9990, revision 1.
  Components: citrus and chilli dressing, slow-roast pepper relish, mussel liquor reduction,
  cultured cream spread.
  Prepared to order and finished on the flat-top; held no longer than four minutes.
  The energy figure is a working estimate and has not been analysed.
  Batch yield 30 portions; serve weight controlled at the section scale.

SPECIFICATION 4 — Spiced Bean Pot (MI-4072)

  Recipe reference RC-8430, revision 5.
  Supplier allergen declarations held for this recipe:
    fermented soya glaze               declares: soya
    crushed new potato                 declares: no allergens
    brown shrimp butter                declares: crustaceans
    mussel liquor reduction            declares: molluscs
  Declarations are on file for all components listed above; none is outstanding.
  Finished under the salamander and rested before plating.
  The energy figure was determined by laboratory analysis of a finished portion.
  Yield: 36 portions per batch at the serve weight recorded on the panel.

SPECIFICATION 5 — Slow-Braised Squash Roll (MI-6065)

  Recipe reference RC-6556, revision 8.
  Supplier allergen declarations held for this recipe:
    laminated flour crust              declares: cereals containing gluten
    ground almond dust                 declares: tree nuts
    shredded winter greens             declares: no allergens
    toasted sesame finish              declares: sesame
    celery salt rub                    DECLARATION OUTSTANDING — not received
    charred sweetcorn kernels          declares: no allergens
  The supplier allergen declaration for one component above has not been received.
  Batched twice a service and held hot in the pass well.
  Batch yield 36 portions; serve weight controlled at the section scale.

SPECIFICATION 6 — Spiced Pulse Ragout (MI-4614)

  Recipe reference RC-8730, revision 4.
  Components: wild rice pilaf, laminated flour crust, egg-washed glaze, herb oil drizzle.
  Prepared to order and finished on the flat-top; held no longer than four minutes.
  Energy is carried across from the supplier's own figure for the base component and has not
  been checked against a finished portion.
  Prepared in batches of 38 portions against the day's forecast.

SPECIFICATION 7 — Toasted Field Wrap (MI-2539)

  Recipe reference RC-5110, revision 1.
  Supplier allergen declarations held for this recipe:
    cultured cream spread              declares: milk
    toasted sesame finish              declares: sesame
    citrus and chilli dressing         declares: no allergens
    shredded winter greens             declares: no allergens
  A supplier allergen declaration is held for every component of this recipe.
  Prepared to order and finished on the flat-top; held no longer than four minutes.
  The energy figure is a working estimate and has not been analysed.
  Batch yield 44 portions; serve weight controlled at the section scale.

DEVELOPMENT RECIPES NOT PUBLISHED IN THIS RELEASE

  Fireside Root Stack (MI-9938) is a development recipe held for the next menu cycle. It is not
  published in this release and this release makes no disclosure for it. Its specification is
  recorded below for the file only.

  Recipe reference RC-5348, development revision.
  Supplier allergen declarations held for this recipe:
    citrus and chilli dressing         declares: no allergens
    black sesame crust                 declares: sesame
    preserved lupin flour              declares: lupin

Raw responseThe raw response

{
  "release_ref": "MR-2019-698",
  "items": [
    {
      "item_code": "MI-9643",
      "item_name": "Golden Barley Broth",
      "published_kcal": 774,
      "published_allergens": "not stated",
      "spec_completeness": "all_declared",
      "allergen_agreement": "published_omits",
      "calorie_basis": "analysed",
      "allergen_support": "unsupported_omission",
      "calorie_support": "supported"
    },
    {
      "item_code": "MI-2681",
      "item_name": "Toasted Seed Loaf",
      "published_kcal": 856,
      "published_allergens": "tree nuts",
      "spec_completeness": "all_declared",
      "allergen_agreement": "published_omits",
      "calorie_basis": "analysed",
      "allergen_support": "unsupported_omission",
      "calorie_support": "supported"
    },
    {
      "item_code": "MI-6309",
      "item_name": "Autumn Grain Salad",
      "published_kcal": 982,
      "published_allergens": "milk, molluscs",
      "spec_completeness": "no_declarations_recorded",
      "allergen_agreement": "nothing_to_compare",
      "calorie_basis": "estimated",
      "allergen_support": "not_determined",
      "calorie_support": "indicative"
    },
    {
      "item_code": "MI-4072",
      "item_name": "Spiced Bean Pot",
      "published_kcal": 286,
      "published_allergens": "not stated",
      "spec_completeness": "all_declared",
      "allergen_agreement": "published_omits",
      "calorie_basis": "analysed",
      "allergen_support": "unsupported_omission",
      "calorie_support": "supported"
    },
    {
      "item_code": "MI-6065",
      "item_name": "Slow-Braised Squash Roll",
      "published_kcal": 962,
      "published_allergens": "tree nuts, cereals containing gluten, sesame",
      "spec_completeness": "declaration_outstanding",
      "allergen_agreement": "agrees",
      "calorie_basis": "silent",
      "allergen_support": "unsupported_incomplete_spec",
      "calorie_support": "not_determined"
    },
    {
      "item_code": "MI-4614",
      "item_name": "Spiced Pulse Ragout",
      "published_kcal": 1106,
      "published_allergens": "egg, cereals containing gluten",
      "spec_completeness": "no_declarations_recorded",
      "allergen_agreement": "nothing_to_compare",
      "calorie_basis": "estimated",
      "allergen_support": "not_determined",
      "calorie_support": "indicative"
    },
    {
      "item_code": "MI-2539",
      "item_name": "Toasted Field Wrap",
      "published_kcal": 646,
      "published_allergens": "sesame, milk",
      "spec_completeness": "all_declared",
      "allergen_agreement": "agrees",
      "calorie_basis": "estimated",
      "allergen_support": "supported",
      "calorie_support": "indicative"
    }
  ]
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check a restaurant menu's allergen claims against its recipes — 50 disclosure packs. One model answered, and every answer was then graded Five different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersFive ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The scored run only. The 6-pack calibration, the four prompt-split calls at max_tokens=1, the single worked-example replay and the one live UI screenshot are 12 further calls that are NOT in this figure and NOT in any denominator on this page. Grading adds nothing — evals/judge.py makes no model call, and neither does the free name-derived floor it is compared against.

Cost driversWhat actually moves the bill

• The fixed prefix — system prompt (967 tokens), disclosure-support rulebook (960) and field schema (825) — is 2,752 of 4,101 tokens on the example call, 67.1 pct, and is byte-identical on every one of the 50 calls. The disclosure pack itself is 1,349 tokens. The bill is the rulebook and the instructions, not the menu.
• Provider-side reasoning. 168,535 of the run's 198,642 output tokens (84.8 pct) were reasoning rather than the JSON worksheet, on a reply that is about 600 tokens of actual content. Reasoning was left at the provider's default and nothing here has measured what turning it off would cost in accuracy.
• Output length scales with the PACK, not with a fixed record — this kit's reply is a LIST, and published_allergens is a free-text field that scales with the list it copies. A four-item pack and a seven-item pack are not the same call: the run's largest reply was 8,732 output tokens and the 6-pack calibration's largest was 5,939.

Your volumeWhat it costs at your volume

Linear in disclosure packs: each call is independent, carries the same fixed prefix and reads one pack. Ten times the releases is ten times the calls and ten times the bill, with no index to rebuild and no cliff, until a pack carries enough items to walk into the output ceiling — which is a per-pack limit and not a volume one.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 menu disclosure packs, one completion call each, one tier. The 6-pack calibration, the worked-example capture, the four prompt-split calls and the live UI screenshot are recorded separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the disclosure pack into addressable sections, pure code — one per published item's specification, plus the header, the published panel, the supplier account record and any decoy block

# Cut a disclosure pack into addressable sections. Pure code — no model, no network.

FIXED_HEADINGS = (
def _marks(text):
def sections(text):
def item_codes(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections are sent, pure code — section names are matched by PREFIX because every pack's Specification sections are named after its own item codes. The three decoy sections are deliberately SENT; the SUPPLIER CONTACT AND ACCOUNT RECORD is mapped by nothing and never leaves the machine

You change it to: map fields to your own pack's headings, including the per-item prefix; unmatched falls back to the whole document. NEVER_SENT is the list of sections no field asks for and which therefore never leave the machine. ⚑ NOTE THE ONE FIELD THAT NEEDS TWO SECTIONS AT ONCE: allergen_agreement is a COMPARISON, so the published panel and the item's own specification both have to be in the call or the question cannot be asked

# Pick which sections of a disclosure pack are sent. Pure code -- the last deterministic step

HEADER = "header"
PANEL = "PUBLISHED DISCLOSURE PANEL — ITEMS IN THIS RELEASE"
DEVELOPMENT = "DEVELOPMENT RECIPES NOT PUBLISHED IN THIS RELEASE"
CARRYOVER = "ITEMS PUBLISHED UNDER AN EARLIER RELEASE"
AMENDMENT = "MENU AMENDMENT — EXECUTED AFTER THE PANEL ABOVE"
SPEC_PREFIX = "Specification "
NEVER_SENT = ("SUPPLIER CONTACT AND ACCOUNT RECORD",)
SECTION_HINTS = {
def _matches(name, want):

src/rulebook.pyrulebook

the two worksheet calls, loaded from data/rulebook.json — a five-step allergen rule with a stopping order and a three-way calorie rule. Data, not code, so a forker can open it and disagree with it

# THE DISCLOSURE-SUPPORT RULEBOOK, loaded from data/rulebook.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
COMPLETENESS = ("all_declared", "declaration_outstanding", "no_declarations_recorded")
AGREEMENTS = ("agrees", "published_omits", "published_adds", "nothing_to_compare")
CALORIE_BASES = ("analysed", "estimated", "silent")
ALLERGEN_CALLS = ("supported", "unsupported_omission", "unsupported_addition",
CALORIE_CALLS = ("supported", "indicative", "not_determined")
def load(path=RULEBOOK_PATH):
R = load()

src/prompt.pyprompt

assemble one call per pack for every published item, with the rulebook RENDERED into it from data/rulebook.json rather than restated in prose — a rule the model has never been shown is not a rule

# Assemble the worksheet prompt. ONE prompt per disclosure pack, every published item in the reply.

SYSTEM = (
RULEBOOK_TEXT = rulebook_block()
def field_schema(pack_fields, item_fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key — plus the pure-code business-condition check downstream: a pack carrying an item whose declarations are not all in, published with a list that agrees with the ones that are, goes back to the kitchen

You change it to: the review rule itself — this kit ships two values (an outstanding declaration and an agreeing published line) and a real review weighs which allergen is outstanding, how the item is prepared and what the kitchen's cross-contact controls are. It is deliberately NOT the same function as the rulebook lookup, so changing WHAT GETS REVIEWED does not change WHAT THE RULEBOOK SAYS

# Extract one disclosure pack's worksheet: segment, select, prompt, one model call, then a

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 16000
def load_fields():
def item_fields(fields=None):
def pack_fields(fields=None):
def load_doc(doc_id):
def documents():

evals/judge.pyjudge

score identification, the extracted cells, the two calls, the determinacy split and the false-assurance direction, all separately, pure code — plus the no-gold consistency diagnostic, which on this kit catches nothing at all and says so

# Score a disclosure-worksheet run. PURE CODE -- gold is exact and every answer is one value, so

def norm(v):
def code_norm(v):
def equal(got, want):
def _rows_of(rec):
def score_identification(records, golds):
CELL_FIELDS = ("item_name", "published_kcal", "published_allergens", "spec_completeness",
def score(fields, records, golds):
CALLS = {"allergen_support": RB.ALLERGEN_CALLS, "calorie_support": RB.CALORIE_CALLS}
CONFIDENT = {c for c in RB.ALLERGEN_CALLS + RB.CALORIE_CALLS if c != "not_determined"}

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3874 input and 3972 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's disclosure packs are entirely synthetic (tools/build_corpus.py, seed 20260822), so no text in them came from an outside party. The kit makes exactly one outbound request per pack — the completion — and performs no write of any kind: no file is modified, no menu is published, no specification is signed off and nothing is notified. The API key is read from .env or the environment and never printed, logged or committed.

Read from the shared .env or the real environment only, never committed — .env is gitignored from the first commit and this repo has never held a credential. src/config.py::sources() reports WHICH file contributed without printing a value, and src/config.py::save() opens the file at 0600 BEFORE writing so it never exists at the default umask. src/app.py redacts the key and the base URL out of any provider error before returning it to the page.

The experimentWe did not attack it — and three of four boundaries hold

Three boundaries hold. Two were confirmed by reading the code: no code path writes, publishes, signs off or notifies, and the kit makes exactly one outbound request per pack. The third was confirmed by an assertion that runs before any spend — evals/check_labels.py checks both that the SUPPLIER CONTACT AND ACCOUNT RECORD exists in the pack and that it is absent from the sent set, because a selector that quietly started sending it would score identically on every number this page publishes. The fourth is unknown: no injection was ever tried. Confirmed by reading the code and by a pre-flight assertion, not by an attack run. 0 trials fired, on 2026-08-22 — no attack run was fired; see redteam.why.

Three of the four boundaries hold, two confirmed by reading the code (no write path, no second outbound call) and one by an assertion that runs before any spend (the withheld section). The fourth — an instruction hidden in a pack — is UNKNOWN, because no attack run exists.

The result0 attack trials, and three of four boundaries checked here hold on evidence — no write path, no second outbound call, and the one section carrying supplier contact details never leaves the machine.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted decoys measure whether a model reads the PUBLISHED PANEL to decide what is published when three specifications point the other way, and whether it declines a call the paperwork does not support. Neither measures whether it obeys an instruction hidden in the same pack. Those are different failures and only the first two are measured here.

HonestyWhat this does not prove

• Whether an indirect prompt injection in a specification body could move a call. No attack run exists.
• Whether the kit behaves safely against a hostile provider — a response body crafted to break the list parsing in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed document, and nothing beyond that was tested.
• Whether a real disclosure pack would carry anything sensitive this kit mishandles beyond the supplier account section. The corpus has no personal data by construction; a real pack carries named suppliers, negotiated component costs and yield figures both sides treat as confidential, and none of that path is exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_kitchen_review fires when the pack carries at least one item whose specification records an OUTSTANDING supplier allergen declaration and whose published allergen line AGREES with the declarations that are held.

src/extract.py::compute(), called by extract() on every pack, by the eval harness on every run and by the free floor's own output too — one function, three callers, so a flag can never mean two different things.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a compliance determination, a clearance, a sign-off or an authorisation. Nothing here determines compliance with anything, clears an item for service, signs off a specification, authorises a menu, posts anything or notifies anybody. allergen_support and calorie_support are proposals with the rulebook's own reasoning attached, and a qualified reviewer decides.
• It is NOT a check on whether the extracted facts are right. If the model misreads a specification and then applies the rulebook correctly to the misreading, the pack is flagged — or not flagged — on a wrong fact, and this rule cannot tell. THAT IS NOT HYPOTHETICAL HERE: all 21 of the run's allergen errors are exactly that shape, and the consistency diagnostic in evals/judge.py, which is the closest thing to that check, caught 0 of them. It caught 0 of the free floor's 138 too.
• It is NOT any business's QA policy. An outstanding declaration paired with an agreeing published line is this kit's own simplification, invented for this corpus. No statute, regulation, labelling law, standard, regulator's guidance, certification scheme or company manual was consulted, and none is reproduced.
• It is NOT anything about cross-contact. A shared fryer, a shared board and a shared scoop are where most real allergen risk lives, and this rule reads two fields off a document that settles none of it.

WatchedWhat is watched, and why that one

The kit measures 16 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 9 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 50, extraction_cells 2250, failures 0, refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

extraction · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on documents, extraction_cells, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

extraction · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 7 chips that all say so.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/extract.py, to data/fields.json's two enum vocabularies, or to the corpus's bucket composition. evals/check_labels.py runs the firing-rate check before any run may spend, so a condition that drifts out of the 10–60 pct band stops the build rather than shipping.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt names nothing, and a line in it would have to be imported by something under src/ or evals/.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per pack — segment, select, one call, parse, span, a pure-code rule — with no branch, no loop, no retry beyond the adapter's own transient backoff, and no state carried between packs.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain — the JSON extraction from a fenced reply, the list parsing, the backoff policy, the section regex, the prefix matcher, the hanging-indent wrapper in the generator and the .env reader are all code somebody has to own.
• No framework means no framework's ecosystem — no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. The reply is a LIST and nothing validates its shape except src/prompt.py::parse, which fails closed to {} rather than to an empty worksheet, because an empty items list is a legitimate answer here.
• And no framework would have caught this kit's actual defect. 21 of 22 errors follow from a contradiction between the system prompt and a field hint — two pieces of English in two files that a structured-output schema would have happily rendered and never compared.
• The provider abstraction covers exactly two wire formats. A third provider is one function and one dict entry, and until somebody writes it the kit runs on two shapes.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-allergen-declare on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b000-allergen-declare-nameline, t000-allergen-declare-stub recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Every kitchen name, menu name, release reference, item code, dish name, component, recipe reference and figure is invented; no real restaurant, group, brand, product, supplier, certification body, manufacturer, person or recipe is named, and no statute, regulation, labelling law, standard or regulator is named either. The allergen vocabulary is a generic list of the kinds of thing a kitchen declares and is NOT any jurisdiction's schedule. The disclosure-support rulebook this kit ships (data/rulebook.json) is ILLUSTRATIVE and is not an authority: it was written for this kit and reproduces no statute, regulation, labelling law, standard, regulator's guidance, certification scheme or company QA manual. It has no view on cross-contact in a kitchen, which is where most real allergen risk lives. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineWhich items reached the worksheet

Did the run list exactly the items this release publishes? Micro precision and recall over item codes, with the two directions named and never averaged: a MISSED item is a published item left off, and a PHANTOM item is a disclosure on the worksheet that this release never made. Phantoms are broken down by which decoy produced them — an item withdrawn by the amendment, a development recipe, an item continuing under an earlier release, or a code that appears nowhere in the pack.

MethodHow the test was run

evals/judge.py::score_identification, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, which records the published items AND the decoy codes for every pack; evals/check_labels.py asserts before any run may spend that every decoy code is in the pack, collides with no published item, and that each of the three kinds clears a floor of 8.

How tight can the band be? No tolerance band — a code is on the worksheet or it is not, compared after whitespace stripping and upper-casing.

Cadence: Re-score on any change to data/rulebook.json's what_is_a_published_item_here, to src/prompt.py's rule 1, or to the corpus's decoy composition.

The decisionWhen to reach for it

In one linePer-(item, field) exact match, light normalisation

Does the model's value for each of the eight per-item fields match gold, after trimming whitespace and punctuation, plus one release reference per pack? THE DENOMINATOR IS GOLD'S ITEMS, NOT THE RUN'S: a gold item the run never returned scores as a miss on all eight of its cells rather than disappearing, so a run cannot raise this number by returning less.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, generated with the corpus and validated by evals/check_labels.py — every value must be readable off the pack it labels, and every published allergen term is checked term by term rather than as one string, because the list is shuffled in the text.

How tight can the band be? Exact match after stripping surrounding whitespace and trailing punctuation, case-insensitive. Numbers compare as their string form, so 612 and "612" are the same cell.

Cadence: Re-score on any change to data/fields.json's hints or to the corpus generator's sentence templates.

The decisionWhen to reach for it

In one lineThe two rulebook calls, five-way and three-way

On the items BOTH sides list, did the run reach the same allergen_support (one of five) and calorie_support (one of three) the shipped rulebook does? Scored on MATCHED items only, deliberately: a call about an item the run never found is already counted once as an identification failure.

MethodHow the test was run

evals/judge.py::score_calls, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/rulebook.py::decide over the three facts the pack states. Gold, the prompt and this grader read the same data/rulebook.json, so they cannot drift about what a call means.

How tight can the band be? Closed-vocabulary equality. A value outside the rulebook's own list is recorded as unanswered rather than as wrong, because 'the reply is not in the vocabulary' and 'the reply picked the wrong option' are different defects.

Cadence: Re-score on any change to data/rulebook.json.

The decisionWhen to reach for it

In one line⚑ False assurance — how often it said supported and should not have

Of every call scored, how many were answered supported where the rulebook does not support them? supported is the only value on either call that takes a row OFF a reviewer's list; every other wrong answer sends it back for another look. This is counted separately from accuracy because it is a DIRECTION, not a rate — a run can be accurate and have every one of its errors pointing this way.

MethodHow the test was run

evals/judge.py::score_calls, the assurance block, in-process.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: gold's own two calls, from the shipped rulebook. missed_assurance — a supported item questioned anyway — is published beside it because the two costs are nothing alike.

How tight can the band be? No band. supported is either the rulebook's answer or it is not.

Cadence: Re-score on every run, and before any change to compute() or the rulebook.

The decisionWhen to reach for it

In one lineneeds_kitchen_review against the same rule run over gold

Does the pure-code business flag fire on exactly the packs where the condition holds? It is computed from the RUN's own two facts and compared against the same function run over GOLD's items, so it measures the whole path and not just the rule.

MethodHow the test was run

evals/judge.py::score_flag, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute over gold's items. An unanswered reply is NOT folded into 'no' — a run that returns nothing must not score as a careful one.

How tight can the band be? Boolean, three-valued: True, False, or None when the reply carries no item the rule can read. None is scored as unanswered, never as a pass.

Cadence: Re-check on any change to compute() in src/extract.py.

The decisionWhen to reach for it