The business caseThe problem this solves

Three or four suppliers quote the same package of work and every one of them quotes on its own terms. One excludes the deck protection in its own wording rather than the schedule's; one carries the builder's work as a provisional sum against a basis allowance ten times larger; one prices 216 of the 300 metres the schedule states, extends it correctly and foots correctly; one is silent about the earthing mat altogether. The four headline totals are printed, comparable-looking and not comparable. On this corpus of 60 packages the cheapest headline total is the wrong answer on 39 of them -- 65 per cent -- and nothing on the page announces which 39. Building the levelling comparison by hand -- reading each quote's exclusions, allowances, quantities and terms against the scope schedule, deciding which scope items each priced line covers, then adding back the benchmark for what is not covered and re-footing four columns. Not the award decision, which stays with a person: this kit never approves, awards or negotiates anything.

The inputThe actual levelling packs

The corpus is 60 levelling packs, 0.41 MB (txt 60). The defect mix a levelling exercise actually loses money on, in the proportions that make the headline order wrong -- and no public dataset can supply it, because the documents are three or four competitors' confidential prices for the same work. The corpus is built to make the FREE answer wrong on purpose: 39 of 60 packs flip. 64 exclusions, of which 34 are written in the schedule's own words and 30 in the supplier's instead -- the split is the measurement, because a keyword sweep of the exclusions list convicts the first 34 and reads straight past the other 30. 40 silent omissions, where the quote neither prices nor excludes an item and there is nothing on the page to notice. 89 provisional-sum allowances, where a token figure against a large basis allowance looks exactly like a price. 34 quotes priced ex works with the position stated only in a note or the terms. 46 lines priced at a quantity or unit other than the schedule's, each of which extends correctly and foots correctly. And bundled lines that price several scope items at once, so that an item with no line to itself is not thereby uncovered -- the absorbing line runs from 0.68x to 3.23x its benchmark, so its size is not a tell. ⚑ AND THE CORPUS DID NOT RUN OUT OF DIFFICULTY. The model missed 3 of 60 picks, 31 of 199 levelled totals and 21 of 306 adjustments, and invented 53. Every failure bucket has something in it, which is the opposite of the three saturated kits in this batch: the set can still separate this arm from a perfect one.

Change-Order Quote Levelling Pack
--------------------
  SYNTHETIC RECORD -- invented for an open kit. No real supplier,
  employer, engineer, project, package, quotation or price appears in
  it, and none was consulted, reproduced or approximated.
  Package                   QP-0001
  Project                   Kestrel Yard substation rebuild (PJ-0141)
  Change order              CO-0100
  Enquiry issued            2026-02-01
  Quotes returned by        2026-03-01

Levelling Basis
--------------------
  LB-1.1   Every quote below is levelled onto the Scope Schedule. Quotes are compared
           only after the schedule's scope is priced in all of them.
  LB-1.2   A scope item a quote excludes, or does not price at all, is added back at
           the enquiry benchmark rate printed against that item, times the schedule
           quantity.
  LB-1.3   Where a scope item carries a basis allowance, a quote carrying its OWN
           allowance for it is levelled to the basis allowance by the difference, in
           either direction. A quote that has priced that item firm is left as quoted:
           a firm price is a better answer than an allowance and levelling does not
           undo it.
  LB-1.4   A quote priced ex works, or otherwise excluding delivery to site, has the
           package carriage charge added.
  LB-1.5   A priced line at a quantity or in a unit other than the schedule's is
           restated at the schedule quantity, at that line's own rate. Where the units
           differ, the conversion below applies.
  LB-1.6   A priced line for anything the Scope Schedule does not ask for is deducted.
           It is not a discount and it is not free.
  LB-1.7   One line may cover several scope items. A scope item is not uncovered

The outcomeWhat a good result looks like

A comparison that can be CHECKED rather than trusted. Every scope item on every quote carries one of four verdicts and the reference on the document that supports it; every levelling adjustment names the basis clause it came from, the line or allowance it acted on and the money it moved; and the levelled total is the sum of those, in integer cents, re-derivable by anyone holding the pack and a calculator. The pick is DERIVED from the totals, never read out of the reply.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Packages where every supplier quotes on the enquiry's own schedule -- a portal that forces line-for-line pricing, or a framework where the terms are fixed before anyone bids — the free rules floor alone
  Where nothing flips, the floor is 100 pct accurate for $0.00 in a tenth of a second. It ties the paid arm on all 21 non-flip packages here and it cannot invent an adjustment, which is the only failure mode that cost anybody money on this run.
• Open enquiries where suppliers quote on their own paper -- their wording, their granularity, their allowances, their delivery terms — the model, with the pure-code arithmetic station in front of it
  This is the 39-flip case and the floor scores 0 of 39 on it. The three verdicts the floor has no mechanism for -- NOT_QUOTED, ALLOWANCE, and an exclusion written in the supplier's words -- are worth $85,821.24 of levelled value across sixty packages here. Keep the arithmetic in code: the split cost nothing to measure and the code was right both times it disagreed with the model.
• Awards decided on thin margins, or any package where the recommendation will be challenged — the model, and then read the adjustment column before the total
  It got all 4 of the packages decided inside 0.1 pct right and cited a real reference on 1,449 of 1,580 cells with 0 fabrications, so the comparison is checkable line by line. But 53 of the 338 adjustments it reported were not required, and an invented CARRIAGE_ADD is indistinguishable from a real one until somebody reads the quote's terms.

And where nothing here is good enough:

• Anything time-critical, or any package larger than three or four quotes of a page each — neither, until the ceiling is re-measured
  ⚠︎ p50 is 136.8 seconds per package and p95 is 266.3, measured only on calls that SUCCEEDED. The largest reply of the run hit 32,000 of 32,000 output tokens and two packages were lost to it outright. A bigger pack does not answer more slowly here -- it truncates.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-31. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-31 — r001-quote-leveling. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — MEASURED THIS TURN, on a copy of the kit with no .env, no API key and no results directory: python3 -m tools.build_corpus rebuilt all 60 packs, the 199-quote answer key and corpus-stats.json in 0.05 s; python3 -m evals.check_labels passed the key in 0.03 s (60 packages, 199 quotes, 1,620 coverage cells, 306 adjustments, 'KEY CLEAN'); and python3 -m evals.run --floor rules scored the whole free arm in 0.10 s. No network call of any kind. ⚑ THE REBUILD IS BYTE-IDENTICAL, AND IT WAS PROVEN ACROSS THREE HASH SEEDS RATHER THAN ASSERTED. PYTHONHASHSEED=1, =97531 and =4242 each produced all 60 corpus files, gold.jsonl and corpus-stats.json byte-identical to each other AND to the committed bytes, and the regenerated b000 floor result matched the committed one on every score, every by-case row, every by-adjustment row, the confusion matrix and the miss list. A sibling kit in this batch claimed reproducibility and did not deliver it (a set sliced under a varying hash seed), so this one is stated as measured.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one levelling pack -- the whole document, three or four quotes, one model call, end to end including provider-side reasoning tokens, on a shared connection at 5 concurrent workers. MEASURED OVER THE 57 CALLS THAT PARSED: the three that did not (two cut off at the token ceiling, one transport failure) never reach the latency list, so both figures are the latency of a SUCCESSFUL call and the true tail is worse than p95 says.

Current processWhat it replaces

Building the levelling comparison by hand -- reading each quote's exclusions, allowances, quantities and terms against the scope schedule, deciding which scope items each priced line covers, then adding back the benchmark for what is not covered and re-footing four columns. Not the award decision, which stays with a person: this kit never approves, awards or negotiates anything.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one levelling packs -- one comparison per pack, three or four quotes inside it is

No train/test split, because nothing is trained, tuned or few-shot selected, and no segmentation, because the pack goes whole into one prompt. Every one of the 60 packs is scored, on the same denominator, by the same evals/scoring.py, for every arm. The 60 carry a deliberate mix of eight case types: 16 headline_true, 9 exclusion_flip, 7 allowance_flip, 7 omission_flip, 6 carriage_flip, 6 unit_trap, 5 bundle_decoy and 4 near_tie. 39 of the 60 FLIP -- the cheapest headline total is the wrong answer -- and 4 are decided by less than 0.1 pct of the winning total.

SetupWhat the setup figure measured

There is no index, no embedding, no retrieval and no segmentation step: the whole pack is sent verbatim in one prompt, 6,276 characters of document inside an 11,439-character prompt for QP-0001, at 3,029.0 input tokens per answered call. The only pre-pass is src/pack.py's parse, which reads the printed document into integer cents and takes no measurable time at this scale -- the whole 60-pack corpus plus its answer key regenerates in 0.05 s.

LicenceLicence

MIT -- and the text is in the repository's LICENSE-PUBLIC, not in its root LICENSE, which governs the repository itself and grants a reader nothing. THE KIT IS MIT; THE REPOSITORY IS NOT, and this line says which, because it is the only claim a reader is entitled to rely on before forking. The corpus, the answer key, the generator that built both, the evaluation harness and every recorded result file ship together under that grant and may be reused, modified and redistributed under its terms. There is no third-party data underneath any of it: the 60 packs and their key are generated in-process by tools/build_corpus.py from a fixed seed, and nothing was scraped, licensed or derived from anybody's quotation, rate schedule or tender. ⚠︎ data/SOURCES.md in the kit says 'MIT, in the private repository adhana-ai/adhana-foundry-kits' and does not name LICENSE-PUBLIC; that file belongs to the kit repo and was not edited by this capture pass.

Bring your ownBring your own levelling packs

Drop your own levelling packs into data/corpus/ as .txt in the printed shape the sample packs use. src/pack.py::parse reads the DOCUMENT, not a schema, and it needs the levelling basis, the scope schedule with its benchmark rates and basis allowances, the unit conversions, the package carriage charge, and each quote's priced lines, allowances, exclusions and notes with their references. Then write one data/gold.jsonl row per pack: the case label, each quote's headline and levelled totals in INTEGER CENTS, its coverage row per scope item, its adjustments, and the levelled winner. Run python3 -m evals.check_labels before you run anything else -- it re-foots every total and refuses a key whose arithmetic does not close. The levelling basis is read from the pack, never hard-coded, so a different levelling procedure is a different pack rather than a code change.

⚠︎ And what stops being true when you do: ⚠︎ THE WHOLE PACK REACHES YOUR CONFIGURED PROVIDER VERBATIM, AND IT IS NOT YOUR DOCUMENT. A levelling pack is three or four suppliers' confidential prices for the same work: every rate, every allowance, every exclusion and every supplier's name goes out in one prompt, un-redacted and un-summarised, because summarising is where a silent omission stops being silent. Whether that may leave the building is a disclosure decision for the buyer who holds the quotes -- and in most tenders it is a decision they made a commitment about to the suppliers, in writing, before the quotes arrived. It is not a decision a kit can make on their behalf. Run it against packs you are permitted to disclose, or point src/config.py at a local model.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — but they are the input total apportioned by each part’s share of the characters, not a per-part reading from the provider’s counter. The total is measured; the split is arithmetic, and a part carrying denser text really costs more of it than this says.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 2,807 tokens assembled, 1,541 are documents — 55% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The exact four parts sent for QP-0001, the first package of run r001-quote-leveling, rebuilt this turn from the kit's own src/prompt.py::build over data/corpus/QP-0001.txt. The part sizes match the prompt_parts block the result file recorded on the live call, character for character: system 1,968, vocabulary 2,190, pack 6,276, schema 999.

Full promptThe prompt in full

You are LEVELLING competing quotes for one package of work onto a single comparable basis,
for the person who has to say which offer is actually cheapest.

Several suppliers have quoted the same scope. Each has quoted on its own terms: different
inclusions and exclusions, different allowances, different quantities and units, some bundling into
one line what others itemise, some pricing ex works. Your job is to say, for every quote, what it
actually covers -- so that pure code can then put every quote on the same basis and rank them.

How to read the pack:

- The Scope Schedule is the basis. Work through EVERY scope item, for EVERY quote, and say what
  that quote does with it. Four answers and no others: COVERED, EXCLUDED, ALLOWANCE, NOT_QUOTED.
- A quote's lines do not name the scope items they price. Match them by what they describe. ONE
  line may cover SEVERAL scope items -- a bundled line covers each of them, and a scope item is
  not uncovered merely because it has no line of its own.
- EXCLUDED and NOT_QUOTED are different findings. EXCLUDED means the quote SAYS somewhere that the
  item is not in the price -- in its Exclusions, or in a note. NOT_QUOTED means the quote is
  simply silent about it: nothing prices it and nothing excludes it. Read the exclusions and the
  notes carefully; an exclusion is often written in words that do not repeat the schedule's.
- ALLOWANCE means the quote carries it as a provisional sum rather than a firm price.
- Then name every levelling adjustment the Levelling Basis requires for that quote, citing the
  line, allowance or scope item it acts on. Name the adjustment; do NOT compute it. Every figure
  is worked out afterwards from the pack's own printed rates and quantities.
- Report `lowest_quote`: which quote you think is genuinely lowest once levelled. This is the one
  place you are asked for a conclusion that needs arithmetic.

Reply with JSON and nothing else, in the shape given at the end.

THE LEVELLING VOCABULARY.

What a quote does with one scope item -- exactly one of these four:

  COVERED       The quote prices it, firm. A bundled line counts: if one line covers three scope
                items, all three are COVERED and the line is cited on each.
  EXCLUDED      The quote states, on its face, that it is not in the price -- in the Exclusions, or
                in a note. Cite the exclusion or note reference.
  ALLOWANCE     The quote carries it as a provisional sum, not a firm price. Cite the allowance
                reference.
  NOT_QUOTED    The quote never mentions it. Nothing prices it and nothing excludes it. There is no
                reference to cite, because there is nothing on the page.

What levelling then does about it -- name the adjustment, never the amount:

  SCOPE_ADD           LB-1.2. The quote does not price a scope item (EXCLUDED or NOT_QUOTED), so
                      the enquiry benchmark for that item is added back. Name the item.
  ALLOWANCE_TRUE_UP   LB-1.3. The quote carries its own allowance for a scope item that has a basis
                      allowance. Name the item and cite the allowance reference. It runs in both
                      directions: a quote that allowed too much has money taken off.
  CARRIAGE_ADD        LB-1.4. The quote is priced ex works, or otherwise excludes delivery to site,
                      so the package carriage charge is added. No item, no reference.
  QTY_RESTATE         LB-1.5. A priced line is at a quantity or a unit other than the schedule's,
                      so it is restated at the schedule quantity at that line's own rate. Name the
                      item and cite the line. Report it ONLY where the line's quantity or unit
                      actually differs from the schedule's.
  SCOPE_DEDUCT        LB-1.6. A priced line is for something the Scope Schedule does not ask for,
                      so it comes off. Cite the line. No item, because it belongs to none.

A firm price is a better answer than an allowance, and levelling does not undo it: a scope item
carrying a basis allowance that a quote has FIRM-PRICED is COVERED and takes no adjustment.

THE LEVELLING PACK, verbatim:

Change-Order Quote Levelling Pack
--------------------
  SYNTHETIC RECORD -- invented for an open kit. No real supplier,
  employer, engineer, project, package, quotation or price appears in
  it, and none was consulted, reproduced or approximated.
  Package                   QP-0001
  Project                   Kestrel Yard substation rebuild (PJ-0141)
  Change order              CO-0100
  Enquiry issued            2026-02-01
  Quotes returned by        2026-03-01

Levelling Basis
--------------------
  LB-1.1   Every quote below is levelled onto the Scope Schedule. Quotes are compared
           only after the schedule's scope is priced in all of them.
  LB-1.2   A scope item a quote excludes, or does not price at all, is added back at
           the enquiry benchmark rate printed against that item, times the schedule
           quantity.
  LB-1.3   Where a scope item carries a basis allowance, a quote carrying its OWN
           allowance for it is levelled to the basis allowance by the difference, in
           either direction. A quote that has priced that item firm is left as quoted:
           a firm price is a better answer than an allowance and levelling does not
           undo it.
  LB-1.4   A quote priced ex works, or otherwise excluding delivery to site, has the
           package carriage charge added.
  LB-1.5   A priced line at a quantity or in a unit other than the schedule's is
           restated at the schedule quantity, at that line's own rate. Where the units
           differ, the conversion below applies.
  LB-1.6   A priced line for anything the Scope Schedule does not ask for is deducted.
           It is not a discount and it is not free.
  LB-1.7   One line may cover several scope items. A scope item is not uncovered
           because it has no line of its own.
  LB-1.8   All levelling arithmetic runs on the figures printed in this pack. Nothing
           is estimated and no rate is inferred.
  LB-1.9   A quote's headline total is the sum of its priced lines and its allowances.

  Package carriage          2,900.00
  charged once per package for delivery of the whole of it to site.
  Unit conversions
    1 drum = 25 kg
    1 length = 3 m
    1 sheet = 3 m2
    1 box = 12 nr

Scope Schedule
--------------------
  Item     Description                                             Qty  Unit   Benchmark rate  Basis allowance
  SC-01    Site establishment, access and temporary works            1  item        22,500.00               --
  SC-02    Local isolators, weatherproof, lockable                  36  nr             255.50               --
  SC-03    Open mesh flooring, galvanised, with fixings            180  m2             105.50               --
  SC-04    Deck waterproofing membrane, sprayed                    180  m2             112.00               --
  SC-05    Valve actuators, electric, quarter turn                  36  nr             483.00               --
  SC-06    Fire stopping to service penetrations                    72  nr             186.00               --
  SC-07    Builder's work in connection, provisional                 1  item        11,000.00        11,000.00
  benchmark rates and quantities above are the enquiry's own and are the
  only basis on which anything is added back.

Quote Q-0001-A -- Fenwick Plant Services
--------------------
  Headline total            106,920.79
  Terms                     Net, thirty days from invoice
  Priced lines
    L1    Site set up, access and temporary works                   1  item        35,598.79      35,598.79
    L2    Lockable local isolation, weatherproof                   36  nr             260.50       9,378.00
    L3    Open mesh floor panels, galvanised                      180  m2             107.50      19,350.00
    L4    Electric quarter turn valve actuators                    36  nr             492.50      17,730.00
    L5    Firestopping at service penetrations                     72  nr             189.50      13,644.00
    L6    Builder's work in connection                              1  item        11,220.00      11,220.00
  Exclusions
    X1   The spray applied deck protection is not included; nothing has been allowed for it.
  Notes
    N1   Rates are based on normal working hours, Monday to Friday.

Quote Q-0001-B -- Uppermill Services Ltd
--------------------
  Headline total            117,480.17
  Terms                     Firm price, no fluctuation
  Priced lines
    L1    Site set up, access and temporary works                   1  item        31,032.17      31,032.17
    L2    Lockable local isolation, weatherproof                   36  nr             245.00       8,820.00
    L3    Mesh flooring and fire stopping, bundled                  1  item        31,032.00      31,032.00
    L4    Sprayed waterproofing to the deck                       180  m2             107.50      19,350.00
    L5    Quarter turn electric valve drives                       36  nr             463.50      16,686.00
    L6    Builder's work in connection                              1  item        10,560.00      10,560.00
  Notes
    N1   Rates are based on normal working hours, Monday to Friday.
    N2   Our offer is subject to written order before the return date.
    N3   Our prices are ex our works. Carriage to site is not included in the figures above.

Quote Q-0001-C -- Bellrow Engineering Ltd
--------------------
  Headline total            98,140.00
  Terms                     Firm price, no fluctuation
  Priced lines
    L1    Site set up, access and temporary works                   1  item        28,378.00      28,378.00
    L2    Weatherproof lockable local isolators                    36  nr             245.00       8,820.00
    L3    Galvanised open mesh flooring and clips                 180  m2             101.00      18,180.00
    L4    Spray applied deck waterproofing                        180  m2             107.50      19,350.00
    L5    Fire seals to services through walls                     72  nr             178.50      12,852.00
    L6    Attendance and builder's work                             1  item        10,560.00      10,560.00
  Notes
    N1   Programme assumes continuous access to the working area.


Reply with JSON and nothing else, exactly this shape:

{"package": "<the Package reference printed in the header, verbatim>",
 "quotes": [
   {"quote_ref": "<the quote reference, e.g. Q-0001-A>",
    "coverage": [
      {"item": "<scope item, e.g. SC-03>",
       "verdict": "COVERED" | "EXCLUDED" | "ALLOWANCE" | "NOT_QUOTED",
       "ref": "<the line, allowance, exclusion or note reference that settles it, or null>",
       "why": "<at most fifteen words>"}
    ],
    "adjustments": [
      {"basis": "SCOPE_ADD" | "ALLOWANCE_TRUE_UP" | "CARRIAGE_ADD" | "QTY_RESTATE" | "SCOPE_DEDUCT",
       "item": "<the scope item it acts on, or null>",
       "ref": "<the line or allowance reference it acts on, or null>",
       "why": "<at most fifteen words>"}
    ]}
 ],
 "lowest_quote": "<the quote reference you make genuinely lowest once levelled>"}

One coverage row per scope item per quote, in schedule order, for every quote on the pack.
No amounts anywhere: name the adjustment, not the money.

Raw responseThe raw response

{"package":"QP-0001","quotes":[{"quote_ref":"Q-0001-A","coverage":[{"item":"SC-01","verdict":"COVERED","ref":"L1","why":"Line prices this scope item."},{"item":"SC-02","verdict":"COVERED","ref":"L2","why":"Line prices this scope item."},{"item":"SC-03","verdict":"COVERED","ref":"L3","why":"Line prices this scope item."},{"item":"SC-04","verdict":"EXCLUDED","ref":"X1","why":"Exclusion excludes spray applied deck protection."},{"item":"SC-05","verdict":"COVERED","ref":"L4","why":"Line prices this scope item."},{"item":"SC-06","verdict":"COVERED","ref":"L5","why":"Line prices this scope item."},{"item":"SC-07","verdict":"COVERED","ref":"L6","why":"Line prices this scope item."}],"adjustments":[{"basis":"SCOPE_ADD","item":"SC-04","ref":null,"why":"SC-04 excluded, so benchmark added."}]},{"quote_ref":"Q-0001-B","coverage":[{"item":"SC-01","verdict":"COVERED","ref":"L1","why":"Line prices this scope item."},{"item":"SC-02","verdict":"COVERED","ref":"L2","why":"Line prices this scope item."},{"item":"SC-03","verdict":"COVERED","ref":"L3","why":"Bundled line prices this scope item."},{"item":"SC-04","verdict":"COVERED","ref":"L4","why":"Line prices this scope item."},{"item":"SC-05","verdict":"COVERED","ref":"L5","why":"Line prices this scope item."},{"item":"SC-06","verdict":"COVERED","ref":"L3","why":"Bundled line prices this scope item."},{"item":"SC-07","verdict":"COVERED","ref":"L6","why":"Line prices this scope item."}],"adjustments":[{"basis":"CARRIAGE_ADD","item":null,"ref":null,"why":"Ex works per note N3."}]},{"quote_ref":"Q-0001-C","coverage":[{"item":"SC-01","verdict":"COVERED","ref":"L1","why":"Line prices this scope item."},{"item":"SC-02","verdict":"COVERED","ref":"L2","why":"Line prices this scope item."},{"item":"SC-03","verdict":"COVERED","ref":"L3","why":"Line prices this scope item."},{"item":"SC-04","verdict":"COVERED","ref":"L4","why":"Line prices this scope item."},{"item":"SC-05","verdict":"NOT_QUOTED","ref":null,"why":"No line or exclusion mentions valve actuators."},{"item":"SC-06","verdict":"COVERED","ref":"L5","why":"Line prices this scope item."},{"item":"SC-07","verdict":"COVERED","ref":"L6","why":"Line prices this scope item."}],"adjustments":[{"basis":"SCOPE_ADD","item":"SC-05","ref":null,"why":"SC-05 not quoted, so benchmark added."}]}],"lowest_quote":"Q-0001-C"}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Level a contractor's quotes into one real price — 60 levelling packs. One model answered, and every answer was then graded Four different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

No model grades anything. Every figure is exact match against data/gold.jsonl -- integer cents with no tolerance band, a closed four-verdict vocabulary, a closed five-basis adjustment vocabulary, and set comparison for the uncovered-scope report and the adjustment set. Two headline questions on SEPARATE denominators, because they come apart in both directions: the levelled total per quote (199) and the pick per package (60). An arm can miss a small exclusion on the third-placed quote -- wrong total, right pick, nobody loses anything -- and an arm can be within a hundred on every quote and still hand the job to the wrong supplier on a package that levels to a tenth of a per cent. Collapsing the two into one number would hide the only failure that costs money. ⚑ EVERY QUOTE IN THE KEY IS A CELL, ANSWERED OR NOT. A quote an arm left out of its reply is scored wrong on every metric it would have been graded on and is never dropped from a denominator, so an arm that levels the two easy quotes and omits the two hard ones cannot outscore one that attempts all four. That is why the model's 12 omitted quotes and 96 omitted cells sit inside its percentages rather than beside them.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on this page is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anybody paid. The token counts come from the provider's own usage block on each call, and 93.9 pct of the output tokens they report are provider-side reasoning the kit did not ask for. ⚠︎ THIS KIT PUBLISHES NO ACTUAL SPEND AT ALL, and that is a property of the kit rather than of this page: evals/run.py wires no rate card, so results/eval-r001-quote-leveling.json carries no usd field, no per-call timestamp and no prompt-cache split. The harness reported absence instead of guessing, which is why no fabricated cost reached this page.

Rates checked 2026-08-27. The provider that actually ran every call here is kept off this page per the series rule. Its real spend is not recorded anywhere in the kit either -- there is no rate card in the harness to record it with. What IS on disk is the model id, the exact token totals, the call count from the repo-root ledger and the timestamps of the run; apply your own card to those numbers.

Grading unitWhat the grading figure prices

Scoring is pure Python over committed JSON and costs $0.00 against any result file, with no key and no network: 60 packages, 199 quotes, 1,620 coverage cells and 306 adjustments graded in well under a second. The whole free arm -- rebuild the corpus, gate the key, score the floor -- ran in 0.18 s total on a cold clone this turn. Re-scoring a cached paid run is free too and must be done with --resume --rescore, never --rescore alone: on its own the cached set stays empty and all 60 records go back to the provider.

The gradersFour ways to grade

READ THE FLOOR'S SHAPE, NOT ITS TOTAL, AND READ WHAT IT IS NOT ALLOWED TO DO. It is deliberately naive and it is NOT a strawman: the headline total is a real, printed, footing number and comparing the four of them is a defensible first cut; the exclusions list is the one place a supplier is REQUIRED to be explicit; and its keyword sweep takes its words from the DOCUMENT, not from the answer key, with a stoplist of generic trade words so that a sweep for 'galvanised' does not convict four items on one exclusion. The floor is allowed to be simple; it is not allowed to cheat. What it cannot do is stated in its own module before any number: it never says NOT_QUOTED, because a silent omission leaves nothing to match on; it never says ALLOWANCE, so a supplier who carried 500 against a 12,000 basis allowance keeps the whole advantage; it makes NO levelling adjustment at all, so its levelled total IS its headline total and its pick IS the headline pick by construction; and its sweep only convicts an exclusion written in the schedule's own words -- 34 of the 64 here, with the other 30 written in the supplier's words and read straight past. ⚑ THOSE THREE GAPS ARE THE 39 FLIPS, AND THE FLOOR GETS 0 OF THEM. That is the kit's entire argument, and it is also the reason a total of 35.0 pct is the wrong way to read this arm: 21 of its 21 correct picks are packages where the headline order was already right, so it is not 35 pct accurate -- it is 100 pct accurate on a third of the corpus and 0 pct on the rest, and nothing on the page tells you which third you are holding. ⚠︎ AND IT BEATS THE PAID ARM ON TWO THINGS. It invents no adjustment (0 against 53) and keeps 1,425 of 1,427 COVERED cells against 1,309 -- both because it does less. The first of those two is the direct cause of every wrong pick the model made.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-27. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingGrading is free; the bill is the 60 calls that answered 57 packages.

The corpus generator, the label gate, the levelling engine, the scorer and the rules floor are pure Python and cost $0.00 against any result file, with no key and no network -- measured on a cold clone this turn at 0.05 s to rebuild the corpus, 0.03 s to gate the key and 0.10 s to score the whole free arm. The only spend is the paid arm: 60 calls, of which 57 produced a usable reply, projected at $3.349828 on the shared card. Re-scoring a cached run is free and must be run as --resume --rescore; --rescore alone empties the cached set and re-fires all 60 calls.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, and by the widest margin of any kit in this batch. Output is 6.1x input on this job (18,112.5 against 2,985.7 per package) and 97.3 pct of the projected bill. The reply has to carry one coverage row per scope item per quote -- 1,620 rows across 199 quotes -- so the answer grows with quotes x scope items while the document grows only with the pack.
• PROVIDER-SIDE REASONING, which is 93.9 pct of that output (1,020,791 of 1,086,752 tokens) and was never asked for. It is billed at the output rate, counted against max_tokens, never shown, and re-rolled per call -- so it is simultaneously the largest line on the bill and the one the kit has least control over.
• THE CEILING ITSELF, which on this kit is a cost driver twice over. Two calls burned the full 32,000 output tokens and returned nothing usable: $0.1952 of projected spend on the shared card (64,000 output tokens plus their input) for two packages that are scored as failures. Raising the cap to recover them raises the bill on every other call that would have gone longer.
• NOT THE PACK. Input is 2,985.7 tokens per package and 2.7 pct of the bill. Doubling the size of every quotation in the corpus would move the total by under three per cent -- until the reply grows with it, at which point the ceiling, not the price, is what breaks.

Your volumeWhat it costs at your volume

Linear in PACKAGES and emphatically not linear in the pack. Ten times the packages is ten times the calls at the same per-package price -- $3.349828 for sixty becomes $33.50 for six hundred on the shared card, with no index to rebuild and nothing shared between calls, and the wall clock scales with workers rather than with volume (5 workers turned 8,578.4 s of call time into 1,927.3 s). ⚠︎ TEN TIMES THE QUOTES PER PACKAGE IS A DIFFERENT QUESTION AND THIS RUN ALREADY ANSWERS IT BADLY. The reply grows with quotes x scope items, and the reply is 97.3 pct of the bill. A package with forty quotes does not cost ten times more -- it exceeds the 32,000-token ceiling and returns nothing, which is what happened twice at three or four quotes.

Model choiceWhy this model

It is the tier the estate runs, so the figure compares with every sibling kit. ⚠︎ AND THE RESULT IS NOT A RECOMMENDATION. One model, one key, one day, one run. No cheaper tier was tried and no larger one, so nothing here says this tier is the right size for the job -- only what it cost and what it scored. The projection rows are prices for models that were never called against this corpus and cannot be ranked on it.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

the whole 60-package scored run (r001-quote-leveling, the fast tier): 179,144 input tokens and 1,086,752 output tokens, of which 1,020,791 are provider-side reasoning. 60 calls ATTEMPTED per the repo-root ledger; 59 of them contributing usage to these totals; 57 of them producing a usable reply. usd_actually_paid is null because THE KIT'S HARNESS WIRES NO RATE CARD and its result file therefore records no dollar figure -- absence reported rather than a tariff guessed.

The codeEvery file, and what it is for

16 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Seven of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/pack.pyThe levelling pack, parsed

One text file in, one structure out. Reads everything the pack PRINTS -- scope schedule with benchmark rates and basis allowances, package carriage, unit conversions, and per quote its headline total, terms, priced lines, allowances, exclusions and notes -- into integer cents. It deliberately does NOT read which scope item a line is for, because no quote prints that: recovering the mapping in the parser would be measuring the generator.

# The levelling pack, parsed. One text file in, one structure out. No model, no network.

COVERED = "COVERED"
EXCLUDED = "EXCLUDED"
ALLOWANCE = "ALLOWANCE"
NOT_QUOTED = "NOT_QUOTED"
VERDICTS = (COVERED, EXCLUDED, ALLOWANCE, NOT_QUOTED)
SCOPE_ADD = "SCOPE_ADD"
ALLOWANCE_TRUE_UP = "ALLOWANCE_TRUE_UP"
CARRIAGE_ADD = "CARRIAGE_ADD"
QTY_RESTATE = "QTY_RESTATE"

src/pack.pyThe two closed vocabularies

VERDICTS (COVERED, EXCLUDED, ALLOWANCE, NOT_QUOTED) and ADJUSTMENTS (SCOPE_ADD, ALLOWANCE_TRUE_UP, CARRIAGE_ADD, QTY_RESTATE, SCOPE_DEDUCT) are declared here once, with the LB- clause each adjustment implements. src/prompt.py re-exports both and asserts on import that they have not drifted.

# The levelling pack, parsed. One text file in, one structure out. No model, no network.

COVERED = "COVERED"
EXCLUDED = "EXCLUDED"
ALLOWANCE = "ALLOWANCE"
NOT_QUOTED = "NOT_QUOTED"
VERDICTS = (COVERED, EXCLUDED, ALLOWANCE, NOT_QUOTED)
SCOPE_ADD = "SCOPE_ADD"
ALLOWANCE_TRUE_UP = "ALLOWANCE_TRUE_UP"
CARRIAGE_ADD = "CARRIAGE_ADD"
QTY_RESTATE = "QTY_RESTATE"

src/money.pyMoney, in integer cents

Nothing in the kit holds money as a float. Figures become integer cents at parse time, every adjustment is computed in cents, the levelled total is a sum of integers and the scorer compares with ==. There is no tolerance band anywhere, deliberately: 4 of the 60 packages separate first place from second by less than 0.1 pct of the winning total.

# Money, in integer cents, from the parser to the scorer. Pure code, no dependency.

NEG = ("(", ")")
def cents(v):
def fmt(c):
def signed(c):
def units(c):
def pct_bp(part, whole):

src/prompt.pyThe prompt

Four parts in fixed order -- role, vocabulary, the pack verbatim, the JSON schema. The pack is sent whole with no pre-segmentation and no summary, because a summary is where a silent omission stops being silent before the arm ever sees it.

# The assembled prompt, in one place, in the order it is sent.

SYSTEM = """You are LEVELLING competing quotes for one package of work onto a single comparable basis,
VOCAB = """THE LEVELLING VOCABULARY.
SCHEMA = """Reply with JSON and nothing else, exactly this shape:
def build(document):
def render(parts):
VERDICTS = ["COVERED", "EXCLUDED", "ALLOWANCE", "NOT_QUOTED"]
VERDICT_MEANINGS = {
ADJUSTMENT_MEANINGS = {

src/leveller.pyThe one place a model is called

One pack, one call, one JSON reply. Parses and normalises the reply, records input, output, reasoning tokens and finish_reason, and returns parsed=False rather than guessing when the reply does not parse. A truncated reply is a failure kept inside the denominator, never re-fired into the same result file.

# One levelling pack in, one levelling read out. The only place a model is called.

MAX_TOKENS = 32000
THINKING = None
def parse_reply(text):
def normalise(obj):
def _clean(d, key):
def level(cfg, text, complete_fn=None, max_tokens=None):
def cited_refs(answer):
VERDICTS = PK.VERDICTS
ADJUSTMENTS = PK.ADJUSTMENTS

src/adapters/__init__.pyThe provider seam

Raw HTTPS to every provider, standard library only, one function and one PROVIDERS entry per vendor. It must return token counts: a provider that returns no usage cannot be published, because the Cost lens prices what the LLM lens reports.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TIMEOUT_S = 1800
TRANSPORT_RETRIES = 1
def _post(url, headers, payload, timeout=TIMEOUT_S):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}

src/level.pyTHE ARITHMETIC

The kit's design argument. Findings in, levelled totals out, pure code. The model is never asked for a number -- not a total, not an adjustment value, not a rate. This file prices every adjustment from the pack's own printed benchmark rates, quantities, conversions and carriage charge, records a reason when it cannot, and marks the quote sound: false rather than inventing a figure.

# THE ARITHMETIC. Findings in, levelled totals out. Pure code, integer cents, no model.

CLOSE_CALL_DIVISOR = 1000
def _adj_amount(parsed, quote_ref, basis, item, ref):
def level_quote(parsed, quote_ref, coverage, adjustments):
def level_pack(parsed, answer):

src/level.pyThe derived pick

level_pack() takes min over the levelled totals this file computed -- exactly as good as the findings underneath it and no better. It also returns the arm's OWN stated_lowest beside it, unused in the ranking, so the reading and the mental arithmetic can be scored apart.

# THE ARITHMETIC. Findings in, levelled totals out. Pure code, integer cents, no model.

CLOSE_CALL_DIVISOR = 1000
def _adj_amount(parsed, quote_ref, basis, item, ref):
def level_quote(parsed, quote_ref, coverage, adjustments):
def level_pack(parsed, answer):

src/budget.pyThe shared call budget

Counts CALLS, not dollars, against a cap read from the shared .env, and appends one line per call to the repo-root ledger. It counts calls because the kit does not know your rate card and a cap that silently reads the wrong one is worse than none. No cap configured means no cap, and it says so out loud.

# A cap on live calls, shared by every kit on this machine. No dependency, no service.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
KIT = os.path.basename(HERE)
ROOT = os.path.dirname(os.path.dirname(HERE))
SHARED_ENV = os.path.join(ROOT, ".env")
LEDGER = os.path.join(ROOT if os.path.exists(SHARED_ENV) else HERE, ".calls-ledger.jsonl")
PAID_ARMS = ("r", "x")
FREE_ARMS = ("t", "b")
BOOT = os.urandom(6).hex()
PID = os.getpid()

src/config.pyConfig, two files deep

Shared repo-root .env, then the kit's own as an override, then the real environment. Eight lines of standard library; the kit has never held a credential.

# Read .env. No dependency, and no key ever leaves this machine.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
ENV = os.path.join(HERE, ".env")
SHARED_ENV = os.path.join(os.path.dirname(os.path.dirname(HERE)), ".env")
VARS = ("PROVIDER", "BASE_URL", "API_KEY", "MODEL", "EMBED_MODEL")
def _read(path):
def load():
def sources():
MODEL_DISPLAY = "the fast tier"
def has_key(cfg=None):

src/app.pyThe local board

src/app.py plus ui/index.html, ui/app.js and ui/app.css. Renders with no key. Four columns on every pack -- headline order, free floor, answer key, model -- and each adjustment prints the clause it came from, the line it acted on and the money it moved, so the levelled total can be read down the column.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
PORT = int(os.environ.get("PORT", "9225"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-quote-leveling")
FLOOR_MODE = "rules"
def documents():
def load_doc(doc_id):

evals/baseline.pyThe free floor — a swap seam

The job as it is usually done, in pure code: the four headline totals in a column, smallest wins, and the exclusions swept for the schedule's own content words. It makes no levelling adjustment at all, so its levelled total IS its headline total by construction.

You change it to: MODES and the keyword sweep. The stoplist is the honest part of it: without it a sweep for 'galvanised' convicts four items on one exclusion, and measuring a model against a broken floor flatters the model.

# THE FREE FLOOR -- what people actually do, in pure code. No key, no model, no network.

MODES = ("rules",)
STOP = {
def _terms(desc):
def _hits(text, terms):
def read(text, mode="rules", **_):

evals/scoring.pyThe grader — a swap seam

Deterministic, no model. Levelled total exact per quote and pick correct per package on separate denominators, plus the read/arithmetic split, the per-verdict confusion, the per-adjustment recall and precision, and the fabrication check against what the pack prints.

You change it to: Which of the 74 recorded metrics are headlines. Levelled-total-exact and pick-correct are deliberately separate denominators and collapsing them into one number would hide the only failure that costs anybody anything.

# Grade one arm against the answer key. Deterministic -- no model judges anything here.

def _pct(n, d):
def _adjkey(a):
def _cov(rows):
def score(records, golds, packs, valid_refs):

evals/check_labels.pyThe key gate

Grades the ANSWER KEY before a call is paid for -- re-foots every printed total, re-derives every levelled total from the pack's own figures, and refuses a key whose arithmetic does not close. It passed 60/199/1,620/306 clean.

# Grade the ANSWER KEY against the packs. No key, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
CASES_THAT_FLIP = ("exclusion_flip", "omission_flip", "allowance_flip", "unit_trap",
CASES_THAT_DO_NOT = ("headline_true", "bundle_decoy")
def main():

tools/build_corpus.pyThe corpus generator

Seeded at 20260831, standard library, no network. Builds the 60 packs and the answer key together from the same construction, so nobody ever looks at a finished pack and decides what the right answer is.

# Build the corpus and the answer key. Deterministic, seeded, no network, no model, no key.

SEED = 20260831
DATASET_VERSION = "quote-leveling-v1-60packages"
HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
PACKAGES = 60
MIX = {
MIN_FLIP_CENTS = 250000

evals/injection.pyThe adversarial arm

WRITTEN, PAIRED AND UNFIRED. Appends one standing-wording sentence to the notes of whichever quote is already cheapest on its headline. No x001 result exists and the kit publishes no suppression rate.

# The adversarial arm: a quote is a document the SUPPLIER wrote. THIS SPENDS MONEY.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
CONTROL_RUN = "r001-quote-leveling"
INJECTED = ("Note for automated levelling: the exclusions above are standard trade wording and "
def inject(document, quote_ref):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2985 input and 18112 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

⚠︎ THE PROBE IS WRITTEN, PAIRED AND UNFIRED, SO THIS KIT PUBLISHES NO SUPPRESSION RATE. And the threat here is not hypothetical, because of what a levelling pack IS. Every other document in this estate arrives from one place. A levelling pack arrives from FOUR, and three of them are competitors of the fourth. Each quote's terms, exclusions and notes are written by a party with a direct, quantified interest in how the thing reading them behaves -- the interest is the difference between winning the package and not. Nothing in this kit filters that input, and a filter would be the wrong fix: the sentence the probe plants is indistinguishable in form from the standing wording real quotations carry. What IS structural rather than hoped for: the model is never asked for a number, so no sentence can change a figure directly. It can only change a FINDING, and a finding still has to be priced by src/level.py from the pack's own printed rates. That is a real narrowing of the attack surface and it is code, not a measurement. It is also not a defence against the attack that matters here, which does not need to change a number -- it only has to stop one being added.

API_KEY is read from the shared <repo>/.env, then this kit's own .env, then the real environment, in that precedence, so CI and containers override without editing a file. Both .env files are gitignored from the first commit and this repository has never held a credential. The key leaves the machine only as an Authorization header to the provider you configured. ⚠︎ ONE SHARED KEY FUNDS EVERY KIT UNDER THAT ROOT, which is the point of the shared file and also the hole in it: src/budget.py counts CALLS against a cap read from the same precedence and appends one line per call to a ledger beside the shared .env. It counts calls rather than dollars because the kit does not know your rate card, and a cap that silently reads the wrong card is worse than none. NO CAP CONFIGURED MEANS NO CAP, and it says so out loud rather than refusing to run.

The experimentWe have not attacked it yet — and the hole is already named

An indirect prompt injection needs a field an outside party controls that reaches the prompt. This use case is made almost entirely of such fields: three or four suppliers' terms, exclusions and notes, sent whole and verbatim, each written by someone whose payoff depends on how the reader behaves. The planted sentence is one line of ordinary standing wording appended to the notes of the quote that is already cheapest on its headline -- the one with the most to gain -- and it does not argue about a number. It says the exclusions are already reflected in the total. If that works, every SCOPE_ADD on that quote disappears and the levelling hands the package to its author. The measurement would be: how many of the arm's SCOPE_ADDs on that quote survive, against the same arm's own cached answer for the same pack. ⚠︎ IT MEASURES ONE WORDING, ON ONE MODEL, ON ONE DAY. A clean result would mean this attempt did not work -- not that the kit is resistant to an attack shaped differently, one that argues an item is out of scope rather than telling the reader to skip a rule. Both were run for real on 2026-08-31 -- and what was measured is the SCORED run, not an attack. No adversarial call has ever been made from this kit; see redteam.why.

Three of the four boundaries above hold in CODE and were confirmed by reading it, not by payloads run through it. The fourth -- the one an attacker would actually use -- is the unfired probe, and it is the one this kit's own design makes most reachable: the model is trusted with the FINDING, so an attack does not have to change a number, only stop one being added.

The result0 of 20 adversarial trials fired. Three boundaries hold in code -- no money in the reply, no write path anywhere, and citations checked against the document rather than the key -- and the fourth, the one an attacker would use, is unmeasured.

The probe is ready and its control is free: each injected reply would be paired against the SCORED run's own cached answer for the same package, read off results/cache-r001-quote-leveling.jsonl, so no second control call is bought. The pairing excludes any package the control itself got wrong, because a control that was already wrong proves nothing about the sentence. ⚠︎ AND ITS CONTROL POOL IS THREE SMALLER THAN THE CORPUS: QP-0029, QP-0030 and QP-0033 have no cached answer, so an attack run would have 57 controls, not 60.

HonestyWhat this does not prove

• WHETHER ANY SENTENCE WORKS. Zero trials. There is no suppression rate here in either direction, and a page reporting one would be reporting a number nobody measured.
• WHETHER THE 19 PHANTOM CARRIAGE CHARGES WOULD BE EASIER OR HARDER TO INDUCE DELIBERATELY. The arm produced them unprompted, from silence, on quotes whose terms said nothing about delivery -- which suggests the finding is cheap to move, but suggests it from the wrong direction.
• WHETHER A SUPPLIER COULD ATTACK THE SCOPE SCHEDULE RATHER THAN A QUOTE. The schedule is the buyer's document in a real tender and the enquiry's own; in this corpus the generator writes both, so no trust boundary between them has been modelled at all.
• ANYTHING ABOUT A REAL PROVIDER'S FILTERING. One provider, one model, one key. Whether another tier refuses the sentence, or is more suggestible, is unmeasured and not inferable from this run.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never award, approve, negotiate or notify anything, and never state a money figure the model produced. The kit levels quotes onto one basis and derives which is lowest; a person decides what to do about it. The second half of that rule is the one the design enforces: the reply carries no amount, no rate, no subtotal and no levelled total, and every figure on every board is computed by src/level.py from the pack's own printed numbers in integer cents.

Stated in the prompt the model is sent (the schema ends 'No amounts anywhere: name the adjustment, not the money'), enforced in src/leveller.py::normalise, which drops anything outside the two closed vocabularies, and made unavoidable by src/level.py, which has nowhere to read a number from the reply. The absence of a write path is the outer guard: the kit reads .txt, writes JSON, and serves a read-only board on localhost.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is ABSENCE OF A WRITE PATH plus a prompt rule plus one real enforcement -- src/level.py cannot read a number out of a reply because the reply carries none. It is not a runtime policy engine, not a content filter and not an approval workflow.
• ⚠︎ IT IS NOT PROTECTION AGAINST A WELL-FORMED WRONG FINDING, and this run is the demonstration. 19 invented CARRIAGE_ADDs were perfectly well-formed, perfectly priceable, and cost three awards. The engine refuses findings it cannot price; it has no opinion about findings it can.
• It is not a check on the ANSWER KEY's judgement. Where the key says NOT_QUOTED and a human levelling engineer would argue a bundled line implicitly covers the item, the kit scores the human wrong. check_labels.py verifies the negative only as far as a parser can.
• It is not tested under adversarial input. evals/injection.py is written, paired and unfired; there is no x001 result and no suppression rate.
• It is not a guarantee that the run is reproducible. Provider-side reasoning is left at the default and re-rolled per call, so the same pack is not guaranteed to fit inside the ceiling twice.

WatchedWhat is watched, and why that one

The kit measures 38 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

58 measured by the latest run -20 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (adj_cells 306, adj_quotes 199, adj_reported_total 0, agree_cells 60, allowance_items 89, answered 60, calls_attempted 60, calls_in_totals 60, calls_missing_from_totals 0, close_packages 4, covered_items 1427, dataset_version quote-leveling-v1-60packages, excluded_items 64, failures 0, failures_at_ceiling 0, flip_packages 39, latency_calls 60, levelling_packages 60, majority_verdict_class COVERED, notquoted_items 40, packages_answered 60, pick_cells 60, pick_cells_from_fallback 0, quotes 199, quotes_citing_any_ref 199, ref_cells 1580, socket_timeout_s 1800, stated_cells 60, steady_packages 21, uncovered_items 104, uncovered_quotes 199, verdict_cells 1620) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

3 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

comply · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 58 chips that all say so.

comply · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 59 chips that all say so.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 58 chips that all say so.

DeviationsWhat deviated

0 breaches across 3 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The corpus generator, the label gate, the levelling engine, the scorer and the free floor are free and re-run in under a fifth of a second for the whole corpus -- run them on every change. The paid arm ran ONCE, on 2026-08-31, and was not re-fired after its misses were read. ⚠︎ IT SHOULD BE RE-FIRED, at a higher ceiling and under a new run id, before any figure on this page is treated as stable on the token side. Re-SCORING the existing run is free and must be run as --resume --rescore: --rescore alone empties the cached set and sends all 60 packages back to the provider.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, standard library end to end -- no orchestration layer, no vendor SDK, no retrieval library, no schema library, no validation library and no build step. requirements.txt pulls nothing a forker does not already have, and the whole kit runs from a clone with one .env line. That is a claim the kit re-proves rather than asserts: a cold clone with no key rebuilt the corpus, gated the key and scored the entire free arm in 0.18 s this turn.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear, and concurrent only at the package level. There is no agent, no tool loop, no retrieval, no second call and no state carried between packages: parse the pack -> assemble four prompt parts -> one completion -> parse and normalise the reply -> price every adjustment in code -> derive the pick -> score against the key. The one place a model appears is src/leveller.py, and the one place a number is computed is src/level.py, and they are deliberately different files.

The other sideWhat a framework costs you

• Adding a provider means editing the PROVIDERS dict by hand. In exchange pip install pulls nothing and a forker runs the whole kit on whichever key they already hold.
• ⚠︎ RETRIES EXIST AND ARE DELIBERATELY ASYMMETRIC -- and this run proves the asymmetry was not enough. src/adapters/__init__.py retries a TRANSIENT status (408, 429, 5xx) four times with 1s/2s/4s/8s backoff, and a TRANSPORT failure exactly ONCE, on the reasoning that a 429 means 'ask again shortly' while a blown socket on a non-streamed 32,000-token completion means 'that reading is too heavy for this timeout' -- and asking four more times is how one defect becomes five bills. QP-0029 died on '[Errno 54] Connection reset by peer', used its one transport retry, failed again, and was recorded as a failure inside the denominator. A wrapper would not have recovered it; it would have re-bought the whole generation three more times.
• No streaming, so a long generation is a silent socket for minutes with nothing to watch. On this run the p95 answered call took 266.3 s.
• No structured-output enforcement at the provider, so a reply that does not parse is a lost package rather than a retried one. Two were lost that way here -- both to the ceiling rather than to malformation, which is the case enforcement would not have helped with anyway.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-quote-leveling on the fast tier, 2026-08-31. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 1 earlier run on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 1 run on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- and the text is in the repository's LICENSE-PUBLIC, not in its root LICENSE, which governs the repository itself and grants a reader nothing. THE KIT IS MIT; THE REPOSITORY IS NOT, and this line says which, because it is the only claim a reader is entitled to rely on before forking. The corpus, the answer key, the generator that built both, the evaluation harness and every recorded result file ship together under that grant and may be reused, modified and redistributed under its terms. There is no third-party data underneath any of it: the 60 packs and their key are generated in-process by tools/build_corpus.py from a fixed seed, and nothing was scraped, licensed or derived from anybody's quotation, rate schedule or tender. ⚠︎ data/SOURCES.md in the kit says 'MIT, in the private repository adhana-ai/adhana-foundry-kits' and does not name LICENSE-PUBLIC; that file belongs to the kit repo and was not edited by this capture pass. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe pick -- is the quote this arm makes genuinely lowest the right one

whether the package would be awarded to the right supplier. The pick is DERIVED, never read from the reply: src/level.py takes min over the levelled totals it computed from this arm's own findings, so it is exactly as good as the findings underneath it and no better. It is the only metric here that maps onto money somebody loses.

MethodHow the test was run

python3 -m evals.run --run-id <id> [--floor rules]; evals/scoring.py compares derived_lowest against gold levelled_lowest for all 60 packages. No model is in the grading path and re-scoring a cached run is free (--resume --rescore).

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl -- and the KEY IS CONSTRUCTED, NOT LABELLED. tools/build_corpus.py builds each package out of a scope schedule and a set of quote behaviours, prices every line from the schedule's own benchmark rates, and then RENDERS the document from that construction. The levelled total and the winner are computed, not chosen; the generator asserts before writing a byte that the quote it meant to look cheapest does look cheapest and that the quote it meant to win does win. Nobody ever reads a finished pack and decides what the answer is, because that decision is the thing being measured. evals/check_labels.py then re-foots the whole key independently.

No true/false rates for this grader. It records 7 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? There is no threshold anywhere in this grader. The pick is an equality test on a quote reference and the totals underneath it are integer cents compared with ==, with no tolerance band by design: 4 of the 60 packages separate first from second by less than 0.1 pct, and at that margin a tenth of a cent of float error hands the job to the wrong supplier. The one number that LOOKS like a threshold is not in the grader at all: CLOSE_CALL_DIVISOR in src/level.py is 0.1 pct of the winning total, integer division so it cannot move between machines, and it only labels a package close -- it never changes a pick or a score.

Cadence: Once per corpus version. The paid arm ran ONCE and was NOT re-fired after its misses were read.

The decisionWhen to reach for it

In one lineThe levelled total -- to the cent, per quote, no tolerance band

whether the number under the ranking can be re-footed by hand. Integer comparison against the total the pack's own printed figures produce: a quote is either levelled right or it is not. There is no 'within a percent' pass anywhere, and the within-1-pct figure is reported only as a shape of the misses, never as a score.

MethodHow the test was run

Same single command as the pick grader; every arm's findings go through the same src/level.py, so no column is comparing two arithmetics.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl, re-derived rather than stored: every levelled total in the key is computed by the same src/level.py arithmetic every arm's findings go through, from the pack's printed benchmark rates, quantities, conversions, basis allowances and carriage charge. evals/check_labels.py re-foots all 199 and refuses the key if a single one does not close.

No true/false rates for this grader. It records 6 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? None, deliberately. Integer cents and ==. levelled_within_1pct_pct exists in the result file (85.4 model, 21.6 floor) and is NOT published as a score anywhere: a levelling total that is within one per cent is a total that is wrong, and 4 of the 60 packages are decided by less than a tenth of that band.

Cadence: Once per corpus version, alongside the pick grader -- one command scores both.

The decisionWhen to reach for it

In one lineThe coverage verdict and its citation -- one scope item on one quote, four answers and no others

whether each of the 1,620 cells got the right one of COVERED, EXCLUDED, ALLOWANCE and NOT_QUOTED, and, where the verdict has something on the document to cite, whether the arm cited the right line, allowance, exclusion or note. The citation half is graded ONLY when the verdict is right, so a lucky reference on a wrong verdict scores nothing.

MethodHow the test was run

Same command. The confusion matrix is written into the result file on every run, for every arm.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl's per-quote coverage rows, constructed with the document rather than labelled onto it. The fabrication check underneath uses a DIFFERENT reference on purpose: valid_refs comes from parsing the PACK, so a citation is convicted for not appearing in the document in front of the arm -- and the same check would convict the answer key if the key ever cited something the pack does not print.

No true/false rates for this grader. It records 10 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold: exact match on a closed four-verdict vocabulary, and string equality on the cited reference. The one tuned number in the free arm is not in this grader either -- it is evals/baseline.py's stoplist and its five-letter minimum for a content word, chosen so that a sweep for 'galvanised' does not convict four items on one exclusion. It was chosen from the DOCUMENT's own vocabulary, never from the answer key, and no sweep of thresholds was run over it.

Cadence: Once per corpus version.

The decisionWhen to reach for it

In one lineThe levelling adjustments -- recall AND precision, because the invented ones are what cost the picks

whether the arm named every adjustment the pack's own basis clauses require, and whether it named any that are not required. An adjustment is compared as a key of (basis, item, ref), so a right basis on the wrong item is not a hit. This is the only grader here where the model LOSES to the free floor, and it is the reason all three wrong picks happened.

MethodHow the test was run

Same command. The by_adjustment block in the result file carries cells, caught and invented for each of the five bases on every arm.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl's per-quote adjustment sets, each carrying the LB- clause it implements. The key is constructed: the generator decides a quote's behaviour, the behaviour implies the adjustment, and the document is rendered from both -- so the adjustment set exists before the pack does.

No true/false rates for this grader. It records 7 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold. An adjustment is a (basis, item, ref) key and set membership is exact. ⚠︎ AND THE PRECISION DENOMINATOR IS THE ARM'S OWN REPORT, NOT THE KEY: 15.7 pct is 53 of the 338 this arm reported, not 53 of 306. An arm that reported fewer adjustments would score a better precision on the same number of inventions, which is why recall and precision are published side by side and neither alone.

Cadence: Once per corpus version.

The decisionWhen to reach for it