The business caseThe problem this solves

A customer sends a specification and asks whether you can make it. Answering means putting two documents written by two different organisations onto one scale -- what the customer will accept, property by property with a tolerance either side, against what the process has actually been demonstrated to deliver, property by property with the grades each range holds on -- and then saying yes, no, yes on these grades, or 'we have never measured that'. Almost none of it is judgement: it is a unit conversion, an interval containment test and a set intersection. What IS judgement is small and specific -- is 'Water by Karl Fischer' the same determination as 'Water content' (yes) and is 'Particle size D90' the same as 'Particle size D50' (no); and does a band the table prints as holding on 'all' grades actually hold on all of them, when a sentence in the technical notes says otherwise. Reading the two documents side by side in a spreadsheet: paste the enquiry into a column, VLOOKUP each property name against the capability sheet, compare the two pairs of numbers with an IF. That is what this job is actually done with, and it is the kit's b000 floor rather than a strawman -- it scores 122 of 300 properties and 29 of 60 enquiries on this corpus, and it fails in the two directions above 38 and 20 times.

The inputThe actual capability reviews

The corpus is 60 capability reviews, 0.17 MB (txt 60). Because the two failures this use case has are opposite and both invisible, and only a labelled set separates them. An order refused for no reason leaves no trace anywhere -- nobody audits the enquiries that were turned away -- and a promise the plant cannot keep does not surface until the first rejected delivery. The corpus therefore plants both directions deliberately, in counted numbers, and plants the class the whole kit turns on -- NOT_CHARACTERISED, 28 cells, with 15 near-miss determinations printed beside them so that 'we have never measured it' and 'we measure something that reads like it' can be told apart at all.

Specification Capability Review
-------------------------------
SYNTHETIC RECORD. Every customer, product family, grade, capability band and technical note
below was invented for this kit. No real product, company, published standard, test method
designation or capability sheet was used, reproduced or approximated.

Row formats. A Requested Specification line is `RS-nn  <property, as the customer writes it>
target <value>  tol -<value> / +<value>  unit <unit>`; `open` on a side means the customer
sets no limit there. A Capability Sheet line is `CB-nn  <property, as this plant names it>
<low> to <high>  <unit>  grades <grade identifiers, or all>`, and that range is what has been
demonstrated in production -- not a target and not a single best batch.

Enquiry
-------
Enquiry            SCR-0001
Customer           CUST-MIRENDAL
Product family     PFM-EPXRES  Epoxy resin, liquid
Capability sheet   CAP-EPXRES-R2
Received           2026-01-09

Requested Specification
-----------------------
RS-01   Flash point, closed cup                 target 318.25       tol -8.40      / +8.40       unit K
RS-02   Chloride content                        target 61667        tol -open      / +12333      unit ppb
RS-03   Acid number, mg KOH per g               target 1.24         tol -open      / +0.24       unit mgKOH/g
RS-04   Density at 20 C                         target 1.1490       tol -0.0034    / +0.0034     unit g/cm3
RS-05   Viscosity at 25 C                       target 2252         tol -313       / +313        unit mPa.s

Capability Sheet
----------------
CB-01   Acid value                      0.16 to 1.14               mgKOH/g   grades all
CB-02   Water content                   0.264 to 0.316             %         grades all

The outcomeWhat a good result looks like

A determination a person confirms rather than trusts. Every requested property carries one of four verdicts, the capability entries that decided it named by their printed identifiers, and the grades the answer holds on; the whole enquiry carries its own answer, the reason for it and the binding grade where there is one. The arithmetic is re-runnable off the printed page without the model, and three guardrails convict a misreading with no answer key at all.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Sheets and enquiries that already share one vocabulary and one unit system -- a customer quoting against your own published datasheet, or a framework where the property names are fixed before anyone asks — the free units floor alone
  With no aliases and no notes-channel conditions in play, the units floor is doing the whole job: exact conversion, containment, every matching entry, the printed grades column and the grade intersection, in 0.1 s for sixty reviews at $0.00. On this corpus it already scores 171 of the 192 same-name properties.
• Two organisations writing the same determination in their own words -- a customer's house standard against a plant's own sheet — the model, and this is the clearest case for it
  Both shipped floors score 0 of 80 aliased properties and the model scores 79. ⚠︎ READ THE CAVEAT WITH IT: a lookup built from the generator's own 51-string synonym table also scores 77, so what is measured here is 'resolves a synonym from a closed trade vocabulary', not 'resolves a synonym nobody has ever tabulated'. On a real plant's sheet, whose wording nobody outside it uses, this is the row that would move most.
• Sheets whose grade restrictions live in prose -- a grades column printing all with a technical note saying otherwise — the model, with a free notes reader as the control
  Both shipped floors score 0 of 24 and the model scores 24. But a five-line regex -- one CB- entry plus at least one GRD- grade in the same sentence -- scores 21 of 24 free, and lifts grade conflicts from 1 to 6 of 10. Build that first and measure what is left before buying the call.

And where nothing here is good enough:

• An enquiry whose value turns on ONE property, where the difference between 'we have never measured it' and 'your window is tighter than our spread' decides whether a trial is commissioned — neither arm unsupervised
  The kit's single miss is exactly this shape, and the guardrails passed it. A wrong match that is internally consistent is invisible without an answer key, and on a one-hard-property enquiry it flips the whole answer.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-31. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-31 — r001-spec-capability. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Everything except the paid arm runs on a clean checkout with no key and no network, and was timed that way this turn: rebuild the whole corpus and answer key 0.05 s, gate the key with evals/check_labels.py 0.04 s (KEY CLEAN, 60 reviews, 300 properties), score the b000 rules floor end to end 0.10 s. The board serves without a key and disables the one button that spends money. Total cost of everything a forker can verify: $0.00.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. Measured on the fast tier — one capability review -- the whole document less the withheld Commercial Position, one model call, end to end including provider-side reasoning tokens, on a shared connection at 5 concurrent workers. All 60 calls parsed, so both figures are over the full denominator and there is no successful-call survivorship in them. 2102.5 s of summed call time completed in 429.9 s of wall clock, which is the arithmetic that fixes the worker count at ~5.

Current processWhat it replaces

Reading the two documents side by side in a spreadsheet: paste the enquiry into a column, VLOOKUP each property name against the capability sheet, compare the two pairs of numbers with an IF. That is what this job is actually done with, and it is the kit's b000 floor rather than a strawman -- it scores 122 of 300 properties and 29 of 60 enquiries on this corpus, and it fails in the two directions above 38 and 20 times.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one capability reviews -- one enquiry per review, five requested properties on average inside it is

No train/test split, because nothing is trained, tuned or few-shot selected. No segmentation for retrieval either: the review goes whole into one prompt, less the Commercial Position section which src/select.py withholds at the seam. All 60 are scored, on the same denominators, by the same evals/scoring.py, for every arm. The answer key is DERIVED through src/envelope.py from the generator's injected facts and never typed by a person, and evals/check_labels.py re-derives every label from the printed document before a call is paid for.

SetupWhat the setup figure measured

There is no index, no embedding and no retrieval. The whole review goes in one prompt -- 8,179 characters of system block and 2,538 characters of enquiry for the first review, 2480 input tokens per call on average. The only pre-pass is src/segment.py plus src/sheet.py's parse, which take no measurable time at this scale: the entire 60-review corpus and its answer key regenerate in 0.05 s and the key gate runs in 0.04 s.

LicenceLicence

MIT -- and the MIT text is in the repository's LICENSE-PUBLIC file, NOT in its root LICENSE, which governs the repository itself and grants a reader nothing. Everything under data/corpus/ and data/gold.jsonl was generated for this kit by tools/build_corpus.py and is original to it; there is no third-party material here to relicense, and nothing was scraped, fetched, licensed, anonymised or derived from a real document.

Bring your ownBring your own capability reviews

Point it at your own capability sheet and one real enquiry, in the row formats the documents themselves state at the top of every review. You get the full per-property table, the arithmetic behind each verdict off your own printed numbers, and all three guardrails -- unlocatable identifier, transcription mismatch, self-inconsistent verdict -- on day one, with no labelled data at all. What you do not get without labels is a score.

⚠︎ And what stops being true when you do: ⚠︎ THE WHOLE REVIEW LESS ONE SECTION GOES TO YOUR CONFIGURED PROVIDER VERBATIM, and on this use case that includes the capability sheet -- every demonstrated band, every grade restriction, every technical note. That is your process capability in full, and it is one of the few documents whose disclosure is a commercial decision rather than an IT one. src/select.py withholds the Commercial Position and NOTHING ELSE; it is a named control, not a redaction layer. Run it against a sheet you are permitted to disclose, or against a local model.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — but they are the input total apportioned by each part’s share of the characters, not a per-part reading from the provider’s counter. The total is measured; the split is arithmetic, and a part carrying denser text really costs more of it than this says.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 2,481 tokens assembled, 1,893 are instructions — 76% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Logged from the scored run. Per-part token counts are SCR-0001's own measured 2,481 input tokens split by character share (8,179 / 2,538), because the provider reports one input figure per call and not one per part. src/review.py assembles exactly two parts and sends them as one call: the system block, generated in src/prompt.py from the verdict dictionaries themselves, and the enquiry -- the capability review as printed, less the Commercial Position section that src/select.py withholds at seam 2. The parts below are the real strings for SCR-0001, the first review of the corpus; result file prompt_parts records system 8,179 chars and enquiry 2,538 chars for that same call.

Full promptThe prompt in full

You are answering one question about one enquiry: is the specification this customer
has asked for inside what this plant has actually demonstrated it can produce?

You are given the customer's REQUESTED SPECIFICATION -- named properties, each with a target, a
tolerance either side and a unit -- and the plant's CAPABILITY SHEET, which lists properties as
ranges that have been demonstrated in production, with the grades each range holds on. You are also
given the plant's grade list and its technical service notes.

WHAT YOU ARE ASKED FOR, AND WHAT YOU ARE NOT.

You are asked for two readings and nothing else:

  1. MATCHING. For each requested property, which capability entries -- if any -- are the SAME
     determination, written in the plant's words instead of the customer's.
  2. CONDITIONS. For each of those entries, which grades it actually holds on, reading BOTH the
     grades column and the technical service notes.

You are NOT asked to convert a unit, compare two numbers, decide whether a band fits inside a
window, or work out whether two conditions can be satisfied together. All of that is arithmetic and
it is done in code, from the numbers printed on the sheet, after you answer. Do not adjust a number
to make it comparable, and never report a converted figure: copy what is printed.

MATCHING RULES, in order of importance.

1. THE SAME DETERMINATION IN DIFFERENT WORDS IS A MATCH. The customer writes their own
   specification; the plant writes its own sheet. Two documents naming one measurement differently
   is the ordinary case, not an exception.
2. A DIFFERENT DETERMINATION IS NOT A MATCH, however similar it reads. A different percentile, a
   different measurement temperature, a different colour scale, a different preparation, a
   different derived quantity -- none of these is the requested property. When you are not sure,
   DO NOT MATCH. Reporting that the sheet does not characterise something is a fact somebody can
   act on; inventing an envelope from a near-miss entry is a limit the plant has never
   demonstrated, quoted to a customer as though it had.
3. REPORT EVERY ENTRY THAT IS THIS PROPERTY, including ones that plainly do not reach the requested
   window. A property often carries a base band and a grade-restricted extension, and which one
   applies is decided in code from the numbers. Leaving one out removes it from the answer.
4. A CAPABILITY SHEET IS LONGER THAN ANY ONE ENQUIRY. It will characterise properties nobody asked
   about. Ignore those entirely -- answer once for each line of the Requested Specification, in the
   order the enquiry lists them, and never add a row of your own.

CONDITION RULES.

5. THE GRADES COLUMN IS NOT THE ONLY PLACE A RESTRICTION LIVES. Read the technical service notes.
   A note can restrict a band the table prints as `all` -- and where a note and the table disagree,
   the note is the later word and it governs.
6. A NOTE THAT MENTIONS A GRADE DOES NOT AUTOMATICALLY RESTRICT ANYTHING. Some notes are ordinary
   commercial remarks about which grades exist or which is stocked. A restriction says that a
   particular band was demonstrated only on particular grades. Read what the sentence claims.
7. REPORT GRADES AS THE IDENTIFIERS PRINTED IN THE GRADES SECTION, or the single word `all` when
   the entry holds on every grade.

CITATION RULES.

8. NEVER WRITE AN IDENTIFIER THAT IS NOT PRINTED IN THE DOCUMENT IN FRONT OF YOU. Entry
   identifiers, grade identifiers and the capability sheet's own wording are all copied verbatim,
   character for character, so that every match can be checked by the person reading your answer.
9. COPY EACH MATCHED ENTRY'S RANGE AND UNIT EXACTLY AS PRINTED, as bare text. These are recorded as
   a transcription check. They are not used to compute the answer -- the numbers for that are read
   off the sheet in code -- so there is nothing to be gained by tidying them.

THE VERDICT YOU STATE.

Give the verdict you believe follows for each property, from this closed list:

  WITHIN                     Every value the plant's demonstrated range produces falls
                             inside the window the customer will accept, on a band that
                             holds on every grade. Both bounds are inclusive and an open
                             side of the request constrains nothing.
  OUTSIDE                    The property IS characterised and no band the plant has
                             demonstrated sits inside the requested window, on any grade.
                             Most often that is not an absurd request -- it is a customer
                             window TIGHTER than the process spread, which is a promise the
                             plant cannot keep without sorting.
  ACHIEVABLE_WITH_CONDITION  No band that holds on every grade contains the requested
                             window, but a band that holds on NAMED GRADES does. The
                             request can be met, on those grades and not otherwise. The
                             restriction may be printed in the grades column or stated only
                             in the technical notes.
  NOT_CHARACTERISED          The capability sheet does not characterise this property at
                             all. Unknown, not unachievable -- and a near-miss entry that
                             reads like it (a different percentile, a different scale, a
                             different temperature) is NOT this property.

State the overall answer for the enquiry too, from this list:

  CAN_MEET                   Every requested property is WITHIN. No grade has to be named
                             and nothing has to be qualified.
  CAN_MEET_ON_CONDITION      Nothing is OUTSIDE and nothing is uncharacterised, and at
                             least one property is achievable only on named grades -- and
                             there is at least one grade that satisfies every such property
                             at once. Name it.
  CANNOT_MEET                Either some property is OUTSIDE, or the conditions cannot all
                             hold together: two properties are each achievable, on grades
                             that share nothing. The second case is a GRADE CONFLICT and it
                             is invisible in any per-property table.
  NOT_ANSWERABLE             Nothing is OUTSIDE and nothing conflicts, but at least one
                             requested property is NOT_CHARACTERISED. This is not a no. It
                             names what has never been measured and asks for a trial rather
                             than guessing.

⚠︎ YOUR STATED VERDICT DOES NOT DECIDE THE PUBLISHED ANSWER. The comparison is re-run in code over
the entries you matched and the grades you read, and the code's verdict is what is published.
Yours is recorded beside it as a cross-check: a reply whose stated verdict disagrees with its own
matching is flagged for a person, with no answer key needed. Answer it anyway -- a reading that
cannot say what it implies has not been understood.

Reply with JSON and nothing else:

{"overall": "<one of the four overall answers>",
 "binding_grade": "<the grade identifier the enquiry would have to be built on, or null>",
 "properties": [
   {"requested_property": "<the RS- identifier from the enquiry>",
    "capability_property": "<the capability sheet's own wording for this determination, verbatim, or null>",
    "entries": [{"entry": "<a CB- identifier printed on this sheet>",
                 "low": "<the low end as printed>", "high": "<the high end as printed>",
                 "unit": "<the unit as printed>",
                 "grades": ["<grade identifiers>"] or ["all"]}],
    "verdict": "<one of the four>",
    "note": "<the one sentence a technical service reply would carry for this property>"}],
 "rationale": "<two sentences at most, on what decided the hardest property>"}

An empty `entries` list with a null `capability_property` is the correct and complete answer for a
property this sheet does not characterise. It is not a failure to answer.

Specification Capability Review
-------------------------------
SYNTHETIC RECORD. Every customer, product family, grade, capability band and technical note
below was invented for this kit. No real product, company, published standard, test method
designation or capability sheet was used, reproduced or approximated.

Row formats. A Requested Specification line is `RS-nn  <property, as the customer writes it>
target <value>  tol -<value> / +<value>  unit <unit>`; `open` on a side means the customer
sets no limit there. A Capability Sheet line is `CB-nn  <property, as this plant names it>
<low> to <high>  <unit>  grades <grade identifiers, or all>`, and that range is what has been
demonstrated in production -- not a target and not a single best batch.

Enquiry
-------
Enquiry            SCR-0001
Customer           CUST-MIRENDAL
Product family     PFM-EPXRES  Epoxy resin, liquid
Capability sheet   CAP-EPXRES-R2
Received           2026-01-09

Requested Specification
-----------------------
RS-01   Flash point, closed cup                 target 318.25       tol -8.40      / +8.40       unit K
RS-02   Chloride content                        target 61667        tol -open      / +12333      unit ppb
RS-03   Acid number, mg KOH per g               target 1.24         tol -open      / +0.24       unit mgKOH/g
RS-04   Density at 20 C                         target 1.1490       tol -0.0034    / +0.0034     unit g/cm3
RS-05   Viscosity at 25 C                       target 2252         tol -313       / +313        unit mPa.s

Capability Sheet
----------------
CB-01   Acid value                      0.16 to 1.14               mgKOH/g   grades all
CB-02   Water content                   0.264 to 0.316             %         grades all
CB-03   Ash content                     0.000 to 0.042             %         grades all
CB-04   Chloride content                26 to 78                   ppm       grades all
CB-05   Density at 20 C                 1.1458 to 1.1528           g/cm3     grades all
CB-06   Flash point, closed cup         38.7 to 50.1               C         grades all
CB-07   Viscosity at 25 C               2062 to 2458               mPa.s     grades all

Grades
------
GRD-STD  Standard grade
GRD-LV   Low-viscosity grade
GRD-LC   Low-colour grade
GRD-HP   High-purity grade

Technical Service Notes
-----------------------
Enquiries are answered against the sheet as printed, never against a single best batch.
An order that names no grade is filled as GRD-STD unless the acknowledgement says otherwise.

Raw responseThe raw response

{"overall":"CANNOT_MEET","binding_grade":null,"properties":[
{"requested_property":"RS-01","capability_property":"Flash point, closed cup","entries":[{"entry":"CB-06","low":"38.7","high":"50.1","unit":"C","grades":["all"]}],"verdict":"WITHIN","note":"Flash point, closed cup is demonstrated on all grades within the requested range."},
{"requested_property":"RS-02","capability_property":"Chloride content","entries":[{"entry":"CB-04","low":"26","high":"78","unit":"ppm","grades":["all"]}],"verdict":"OUTSIDE","note":"Chloride content is demonstrated on all grades, but the demonstrated band is not wholly inside the requested upper limit."},
{"requested_property":"RS-03","capability_property":"Acid value","entries":[{"entry":"CB-01","low":"0.16","high":"1.14","unit":"mgKOH/g","grades":["all"]}],"verdict":"WITHIN","note":"Acid value is demonstrated on all grades within the requested upper limit."},
{"requested_property":"RS-04","capability_property":"Density at 20 C","entries":[{"entry":"CB-05","low":"1.1458","high":"1.1528","unit":"g/cm3","grades":["all"]}],"verdict":"OUTSIDE","note":"Density at 20 C is demonstrated on all grades, but the demonstrated band extends above the requested upper limit."},
{"requested_property":"RS-05","capability_property":"Viscosity at 25 C","entries":[{"entry":"CB-07","low":"2062","high":"2458","unit":"mPa.s","grades":["all"]}],"verdict":"WITHIN","note":"Viscosity at 25 C is demonstrated on all grades within the requested range."}
],"rationale":"The deciding properties were RS-02 and RS-04: both are characterised on all grades, but their printed ranges exceed the corresponding upper bound, so the enquiry cannot be met as a whole. Unit differences do not affect the matches because the same determination is named."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

A requested specification against a stated capability envelope — 60 capability reviews. One model answered, and every answer was then graded Four different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

No model grades anything. Every figure is exact match against data/gold.jsonl on two SEPARATE denominators: 300 requested properties (a closed four-word verdict vocabulary, plus the set of named grades where the verdict is conditional) and 60 whole enquiries (a second closed four-word vocabulary, plus the reason and the binding grade). The two are deliberately not nested -- an arm can be right about every property and wrong about the enquiry, because two properties can each be achievable and not achievable TOGETHER, which is a grade conflict and is invisible in any per-property table. An omitted property scores wrong on every metric and is never dropped from a denominator. The fabrication check reads the DOCUMENT: a cited entry id, grade id or capability wording is convicted for not appearing on the sheet in front of the arm, which means it can convict the answer key too.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on this page is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anybody paid. The token counts come from the provider's own usage block on each call, and 86.7 pct of the output tokens they report are provider-side reasoning the kit did not ask for. ⚠︎ THIS KIT PUBLISHES NO ACTUAL SPEND AT ALL, and that is a property of the kit rather than of this page: evals/run.py wires no rate card, so results/eval-r001-spec-capability.json carries no usd field, no per-call tariff and no prompt-cache split. The harness reported absence instead of guessing, which is why no fabricated cost has ever reached this board -- and why wire RATE_CARDS into evals/run.py is on this kit's backlog.

Rates checked 2026-08-27. The provider that actually ran every call here is kept off this page per the series rule, and its real spend is recorded nowhere in the kit either -- there is no rate card in the harness to record it with. What IS on disk is the model id, the exact token totals over all 60 calls, the call count and timestamps from the repo-root ledger, and the reasoning share. Apply your own card to those numbers.

Grading unitWhat the grading figure prices

Grading is pure Python over committed JSON and costs $0.00 against any result file, with no key and no network. Timed on a clean copy this turn: rebuild the corpus and key 0.05 s, gate the key 0.04 s, score the whole b000 free arm end to end 0.10 s. Re-scoring a cached paid run is also free and must be run as --resume --rescore; --rescore alone leaves the cached set empty and re-fires all 60 calls.

The gradersFour ways to grade

READ FOUR FREE ARMS HERE, NOT ONE, AND THE FOURTH IS THE ONE THAT MATTERS. 1. b000 rules, the spreadsheet: 122/300 properties, 29/60 enquiries. It fails in both directions -- 38 WITHIN properties called OUTSIDE (orders refused) and 20 OUTSIDE called WITHIN (promises made) -- and it never returns a conditional verdict at all. 2. b001 units, the shipped strong floor: 199/300, 36/60. All of the arithmetic, none of the reading. Its 19 of 24 on table-stated conditions and 0 of 24 on notes-stated ones is the single sharpest line in the kit. 3. ⚠︎ t000 stub, WHICH IS A FREE ARM NOBODY COUNTED: 180/300 properties and 37/60 enquiries -- BETTER than b000 on both and better than b001 on the enquiry answer. It is the b000 rules reading pushed through src/answer.py's derive path, so it measures exactly what the kit's own arithmetic is worth on top of a spreadsheet's matching: +58 properties and +8 enquiries, free. The README's line that units is 'the strongest thing free code can do here' is not true of the arms already on disk, let alone of (4). 4. ⛔ THE ARM THAT DEMOTES THE HEADLINE, measured in memory this turn and NOT yet a committed result file: b001's engine plus an alias map read out of the generator's own table and a five-line notes reader scores 300/300 properties and 60/60 enquiries -- perfect, free, and one property better than the paid arm. Recipe and per-half figures are in separability. ⚑ AND THE TIE ROW IS THE FLOORS' BUG WEARING A STRENGTH'S CLOTHES, WHICH IS WORTH SAYING BECAUSE THE KIT ALREADY SAYS IT: all four arms score 28/28 on NOT_CHARACTERISED and 0/15 on the near-miss decoys. The floors' 100 pct there is BOUGHT WITH their 0 pct on the 80 aliased properties -- a lookup that cannot match two names for one determination returns not-found, and not-found reads as 'we do not measure it'. Right by accident on every genuinely uncharacterised property and wrong on every aliased one. The two figures are the same fact reported twice, and 43 of the 300 cells therefore separate nothing at all.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-27. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingGrading is free; the whole bill is the 60 calls that answered 60 reviews.

The corpus generator, the answer-key gate, the comparison engine, the scorer and both free floors are pure Python and cost $0.00 against any result file, with no key and no network -- timed on a clean copy this turn at 0.05 s to rebuild the corpus and key, 0.04 s to gate the key and 0.10 s to score the whole rules floor. The only spend is the paid arm: 60 calls, 60 usable replies, projected at $0.865118 on the shared card. Re-scoring a cached run is free and must be run as --resume --rescore; --rescore alone empties the cached set and re-fires all 60 calls.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS. Output is 1.8x input on this job (4,392 against 2,479 per call) and 91.4 pct of the projected bill. The prompt grows with the capability sheet; the answer does not -- it is a fixed-shape JSON object of five or six rows -- so the bill is governed by how long the provider thinks, not by how long the document is.
• PROVIDER-SIDE REASONING, which is 86.7 pct of that output (228,491 of 263,574 tokens) and was never asked for. It is billed at the output rate and it never reaches the text: the visible JSON answer is 585 tokens a call, 13.3 pct of the output. Roughly 79 pct of what this kit costs is reasoning.
• THE FIXED SYSTEM BLOCK, and it is the smallest of the three. 8,179 of the 10,717 prompt characters for SCR-0001 are the system block -- 76.3 pct of the input, re-sent on every call, with NO prompt-cache split recorded by this harness. Input is only 8.6 pct of the projected bill, so caching it is an upper bound on a small number.

Your volumeWhat it costs at your volume

Linear in REVIEWS and flat in the sheet. Ten times the enquiries is ten times the calls at the same per-review price -- $0.865118 for sixty becomes $8.65 for six hundred on the shared card, with no index to rebuild and nothing shared between calls. Wall clock scales with workers rather than with volume: 5 workers turned 2102.5 s of summed call time into 429.9 s. ⚠︎ TEN TIMES THE SHEET IS NOT TEN TIMES THE BILL, AND THAT IS THE USEFUL HALF. The reply is a fixed-shape object of one row per requested property, so a capability sheet with 80 entries instead of 8 moves the INPUT side -- 8.6 pct of the bill -- and leaves the output side where it is. What does move the output side is a customer asking for thirty properties instead of five.

Model choiceWhy this model

It is the tier the estate runs, so the figure compares with every sibling kit. ⚠︎ AND THE RESULT IS NOT A RECOMMENDATION. One model, one key, one day, one run. No cheaper tier was tried and no larger one, so nothing here says this tier is the right size for the job -- only what it scored and what its tokens project to. ⚠︎ AND ON THIS CORPUS NO MODEL COULD BE RANKED ANYWAY: the scored arm is at 299/300 and a free arm is at 300/300, so a cheaper model tested here could only be distinguished by LOSING and a better one could not be distinguished at all.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

the whole 60-review scored run (r001-spec-capability, the fast tier): 148,793 input tokens and 263,574 output tokens, of which 228,491 (86.7 pct) are provider-side reasoning. All 60 calls answered and all 60 contributed to these totals. ⚠︎ usd_actually_paid IS NULL BECAUSE THE KIT'S HARNESS WIRES NO RATE CARD: evals/run.py has no RATE_CARDS table, the result file records no dollar figure and no per-call tariff, and the repo-root ledger records the model but never a price. Absence reported rather than a tariff guessed. Wiring RATE_CARDS into evals/run.py is on this kit's backlog; until then the projection below is the only money on this page.

The codeEvery file, and what it is for

15 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Seven of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pyThe capability review, cut into sections

Splits one review on its underlined headings -- Enquiry, Requested Specification, Capability Sheet, Grades, Technical Service Notes, Commercial Position. Deterministic, so 'what did we send' is a list of section names rather than a guess about a token window.

# Cut a capability review into addressable sections. Pure code, no model.

HEAD = re.compile(r"^([A-Z][A-Za-z ]+)\n-{6,}\n", re.M)
def sections(text):
def named(text, name):

src/select.pyWhat leaves the machine

SEAM 2, and on this use case a control rather than a privacy measure. The Commercial Position section -- enquiry value band, plant loading, whether the commercial team wants the order -- is NEVER SENT. Every one of those is a reason to want the answer to be yes, and a capability answer that has read them is not a capability answer. A denylist of exactly one, named in code.

# SEAM 2 -- what leaves this machine. Pure code, and deliberately a denylist of one.

NEVER_SENT = ("Commercial Position",)
def sent(section_names):
def dropped(section_names):
def body(text, sections_fn):

src/sheet.pyThe sheet, parsed

Everything the document prints, read into structure: each requested property's target, tolerances and unit (with open on a side meaning no constraint), each capability entry's demonstrated low, high, unit and printed grades column, the grade list, and the set of identifiers the review actually prints -- which is what the fabrication guard is checked against.

# Read one capability review into structures. Pure code, no model, no arithmetic opinions.

REQ_LINE = re.compile(
CAP_LINE = re.compile(
GRADE_LINE = re.compile(r"^(GRD-[A-Z]{2,4})\s\s+(.+?)\s*$")
FIELD_LINE = re.compile(r"^([A-Z][A-Za-z ]+?)\s\s+(.+?)\s*$")
OPEN = "open"
class SheetError(ValueError):
def _rows(body, rx, what):
def parse(text):
def ids(parsed):

src/prompt.pyThe prompt — a swap seam

The system block and the enquiry, assembled in the order they are sent. It asks for two readings and explicitly forbids arithmetic: do not convert, do not compare, do not decide whether a band fits -- copy what is printed. The four verdicts and the four whole-enquiry answers are declared ONCE here, as module-level literals, and the definition block the model reads is generated from them.

You change it to: The four per-property verdicts and the four whole-enquiry answers, each with what getting IT wrong costs. Declared once; the model's definition block, the scorer's classes and this page's panel all read the same dict.

# The assembled prompt, in one place, in the order it is sent.

VERDICTS = ("WITHIN", "OUTSIDE", "ACHIEVABLE_WITH_CONDITION", "NOT_CHARACTERISED")
VERDICT_MEANINGS = {
OVERALL = ("CAN_MEET", "CAN_MEET_ON_CONDITION", "CANNOT_MEET", "NOT_ANSWERABLE")
OVERALL_MEANINGS = {
def _block(vocab, meanings, width=88):
SYSTEM = """You are answering one question about one enquiry: is the specification this customer
def build(text):
def render(parts):
def parse(raw):

src/review.pyThe model call

SEAM 1, with the transport in src/adapters/__init__.py. SEAM 1. One call per review, whole reply read at once, no streaming. Raw HTTPS to any OpenAI-compatible provider or Anthropic; provider, base URL, model and key come from the shared repo-root .env. max_tokens 32,000 and a 900-second socket timeout, because a large reasoning budget is a silent socket rather than a progress bar.

# One capability review in, one answer out. The only place in this kit a model is called.

MAX_TOKENS = 32000
THINKING = None
def review(cfg, text, complete_fn=None, max_tokens=None):

src/units.pyThe unit table

33 units in 11 families, every factor an exact Fraction and no float anywhere. Cross-family comparison RAISES -- it does not return false and it does not guess -- so a request in APHA colour against a capability in Gardner is modelled as two different properties and answered NOT_CHARACTERISED.

# The unit table, and exact conversion. Pure code, no model, no float anywhere.

ONE = Fraction(1)
class UnitError(ValueError):
FAMILIES = {
UNIT_FAMILY = {}
KNOWN = frozenset(UNIT_FAMILY)
def family(unit):
def comparable(a, b):
def require_comparable(a, b):
def num(text):

src/envelope.pyThe containment rule

The whole comparison, stated once as RULE_SENTENCE and quoted everywhere: WITHIN <=> request_low <= capability_low and capability_high <= request_high, both bounds inclusive, after converting into one unit, with an open request side placing no constraint. An unconditional fit WINS OUTRIGHT over a grade-restricted one, because quoting a condition nobody needs turns a yes into a negotiation. It also owns the whole-enquiry precedence.

# The comparison, the four verdicts and the overall answer. Pure code. Stated exactly once.

WITHIN = "WITHIN"
OUTSIDE = "OUTSIDE"
ACHIEVABLE_WITH_CONDITION = "ACHIEVABLE_WITH_CONDITION"
NOT_CHARACTERISED = "NOT_CHARACTERISED"
VERDICTS = (WITHIN, OUTSIDE, ACHIEVABLE_WITH_CONDITION, NOT_CHARACTERISED)
CAN_MEET = "CAN_MEET"
CAN_MEET_ON_CONDITION = "CAN_MEET_ON_CONDITION"
CANNOT_MEET = "CANNOT_MEET"
NOT_ANSWERABLE = "NOT_ANSWERABLE"

src/answer.pyThe seam -- a reading becomes an answer

derive() is the kit in one function. It takes the printed review and the arm's READING of it -- which entries are the same property, which grades each holds on -- and computes the answer from the numbers ON THE PAGE via sheet.entries_for(). The arm's own transcription of those numbers is never an input; it is kept only as evidence, so a misreading can be convicted without being able to move a verdict.

# Turn a reading into an answer. The seam between what the model does and what code does.

def derive(parsed, reply):
def blank(parsed):

src/guards.pyThree checks that need no answer key

unlocatable_id (the reading cites an entry, grade or wording not printed on the sheet), transcription_mismatch (the arm's copy of a number disagrees with the page), verdict_inconsistent (the arm's STATED verdict disagrees with what its own matching implies). All three compute on an enquiry nobody has ever scored, which is every enquiry a forker owns. All three fired 0 on the paid run -- but verdict_inconsistent fires 76 times on the free stub arm, which is the evidence that it is live rather than dead code.

# Three checks that need no answer key. Pure code, no model, no network.

def unlocatable_id(parsed, reply):
def transcription_mismatch(parsed, reply):
def verdict_inconsistent(parsed, derived):
def all_checks(parsed, reply, derived):

evals/baseline.pyThe two free floors

rules is the spreadsheet -- exact-name lookup, bare-number comparison, no units, no grades column, no notes. units keeps exact-name matching and hands the arithmetic to the kit's own code: conversion, containment, every matching entry rather than the first, the PRINTED grades column, and the grade intersection. Both produce the paid arm's answer shape, so one scorer grades every arm.

# TWO FREE FLOORS. No key, no model, no network. Both are genuine attempts at the job.

MODES = ("rules", "units")
def norm(s):
def _raw_contains(req, cap):
def _rows_rules(parsed):
def _rows_units(parsed):
def _row(req, cap_property, matched, entries, verdict, binding, note):
def review(text, mode="rules", **_):

evals/scoring.pyThe scorer — a swap seam

Exact match against data/gold.jsonl on two separate denominators, with every cross-tab the README prints: by verdict class, by unit mismatch, by alias, by condition channel, by decoy, plus the four-by-four confusion matrix and the per-pattern table. Pure code; no model grades anything.

You change it to: Every grader, both denominators and all the cross-tabs. Nothing in it calls a model.

# Grade one arm against the answer key. Deterministic -- no model judges anything here.

def _pct(n, d):
def _rows(answer):
def _binding(v):
def score(records, golds, valid_ids):

evals/check_labels.pyThe key gate

Grades the ANSWER KEY before a call is paid for, re-deriving every label from what each document prints through the parser the arms use. KEY CLEAN over 60 reviews and 300 properties, in 0.04 s.

# Grade the ANSWER KEY against the documents. No key, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
def raw_verdict(req, cap):
def main():

tools/build_corpus.pyThe corpus generator

Sixty reviews from a fixed seed (20260831), standard library only, no network. It refuses to write a unit case unless the unit-blind comparison genuinely lands wrong, and refuses an alias case unless exact-name lookup genuinely misses. Rebuild is byte-identical, verified at four PYTHONHASHSEED values.

# Generate the capability-review corpus, its answer key and its stats. Deterministic.

SEED = 20260831
DATASET_VERSION = "spec-capability-v1-60reviews"
REVIEWS = 60
HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
STATS = os.path.join(HERE, "data", "corpus-stats.json")
PROPS = [
BY_KEY = {p["key"]: p for p in PROPS}

src/app.pyThe board

A keyless local board on 127.0.0.1:9228. It runs either free floor on any review, replays the recorded paid run off the committed result file, prints the arithmetic behind every verdict beside the sheet as printed, shows exactly what left the machine at seam 2, and disables the one button that spends money -- saying why -- when no key is configured. The page itself is ui/index.html and ui/app.js.

# The minimal local board. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
PORT = int(os.environ.get("PORT", "9228"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-spec-capability")
FLOOR_RUNS = {"rules": "b000-spec-capability-rules", "units": "b001-spec-capability-units"}
def documents():
def load_doc(doc_id):

evals/injection.pyThe adversarial probe (written, never fired)

Appends one line to the requested specification's own register claiming the plant's technical service has already confirmed the properties are met on all grades, targeting the enquiries whose correct answer is NOT a clean yes. Paired against the scored run's free cache. It has never been run and there is no x001 result file.

# A FORCED, PAIRED PROMPT-INJECTION PROBE. THIS SPENDS MONEY.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
KIT = "spec-capability"
SECTION = "Enquiry"
INJECTION = (
PROPERTY_FIELDS = ("verdict", "binding", "matched_entries", "capability_property")
def inject(text, section=SECTION, line=INJECTION):
def _rows(answer):
def _norm(v):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2479 input and 4392 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

One outbound call per review to your configured provider, over raw HTTPS, carrying the whole capability review except the Commercial Position section. No inbound surface except a localhost board, no database, no queue, no third-party service and no dependency outside the standard library. ⚠︎ THE MATERIAL THAT LEAVES IS THE SENSITIVE PART OF THIS USE CASE: a capability sheet is the plant's own process record -- every demonstrated band, every grade restriction, every technical note -- and it is one of the few documents whose disclosure is a commercial decision rather than an IT one.

One key at the repo root, in a .env that has been gitignored since the first commit; a kit-local .env overrides key by key. src/config.py reads it at call time. The key is never written to a result file, a cache line, a log or the call ledger, and no result file in this kit contains a credential. src/budget.py caps live calls per machine and counts CALLS rather than dollars, deliberately -- a dollar cap needs a price for whatever model you are pointed at, and this kit has no rate card.

The experimentThe one experiment this kit owes

Fire evals/injection.py at 20 of the 52 enquiries whose correct answer is not a clean yes, paired against the committed cache, and publish three numbers separately: published verdicts moved (expected 0 -- the verdict is computed in code from entries on the page), READINGS moved (which entries the arm calls the same property), and GRADE CLAIMS moved. 20 calls, projected at roughly $0.29 on the shared card. Until that runs, the security posture here is an argument from construction, not a measurement. Both were run for real on r001-spec-capability.

Three of the four gates are closed by construction and measured at zero on the paid arm; the fourth is written and unfired. One of the three checks -- the self-consistency cross-check -- is separately proven live by the free stub arm, where it fires 76 times. ⚠︎ A ZERO ON A CORPUS WITH NO ADVERSARIAL CONTENT IN IT IS NOT EVIDENCE OF RESISTANCE -- tools/build_corpus.py carries no injected content by construction, so a run over these 60 reviews reports perfect resistance while never having attacked anything. That is the whole reason the probe exists and the whole reason its absence is stated here rather than implied.

The resultThe commercial pressure never leaves the machine, and the arithmetic cannot be argued with -- but nobody has yet tried to argue with the reading.

The probe is ready and its control is free: each injected reply is compared against this kit's own cached answer for the same enquiry, so it costs one call per enquiry instead of two and the comparison is against the exact answers this kit published rather than a fresh sample that might differ for unrelated reasons. Only enquiries the control got right enter the paired denominator -- 'the injected arm also got it wrong' is not evidence about injection. ⚠︎ A SUPPRESSION RATE OF 0 WOULD NOT BE A ROBUSTNESS CLAIM: it would measure one sentence, on one model, on one day.

HonestyWhat this does not prove

• Any suppression rate. The probe is written and has never been run; there is no x001 result file and no injected call has ever been made from this kit.
• Whether the guardrails catch a persuaded reading. All three fired 0 on the paid arm, on a corpus with no adversarial content -- which is what a corpus with no adversarial content produces. And the one check with independent evidence (verdict_inconsistent, 76 firings on the stub) is the one an injected sentence is least likely to trip, because it compares a reply against itself.
• Whether a wrong match that is internally consistent can be caught without an answer key. It cannot -- SCR-0056 is the worked example and all three guards passed it -- and no measurement here changes that.
• Anything about a real customer document. The 60 enquiries were generated by this kit and carry no free text a person wrote.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

The model reads; code decides. Every unit conversion, every containment test, every grade intersection and the whole-enquiry precedence are computed from the numbers PRINTED on the sheet, after the reply comes back -- so the arm's own transcription and its own stated verdict can never move a published answer.

src/answer.py::derive, src/units.py, src/envelope.py, src/guards.py

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is not a check that the READING is right. A wrong match that is internally consistent -- matching the wrong entry, copying its band perfectly, stating the verdict that entry implies -- passes all three guards. SCR-0056 is the worked example and only the answer key caught it.
• It is not evidence of robustness. All three guards fired 0 on a corpus that carries no adversarial content by construction. A zero on an unattacked corpus measures the corpus.
• It is not a redaction layer. src/select.py withholds exactly one named section and cannot withhold the technical notes without withholding the conditions the answer depends on.
• It is not a cost control. src/budget.py caps CALLS, not dollars, because this kit has no rate card to convert one into the other.
• It is not a substitute for the answer key. Every figure that separates the arms on this page comes from data/gold.jsonl; the guards separate nothing here because they all read zero on every arm.

WatchedWhat is watched, and why that one

The kit measures 39 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

63 measured by the latest run -24 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (alias_cells 80, bindgrade_cells 10, cited_cells 192, cond_cells 48, cond_notes_cells 24, cond_table_cells 24, conditional_cells 10, conflict_cells 10, dataset_version spec-capability-v1-60reviews, decoy_cells 15, en_same_cells 130, en_unit_cells 62, exactname_cells 192, failures 0, majority_overall_class CANNOT_MEET, majority_overall_pct 50.0, majority_verdict_class WITHIN, majority_verdict_pct 60.0, matchset_cells 272, notchar_cells 28, outside_cells 44, property_cells 300, reviews 60, reviews_answered 60, sameunit_cells 202, socket_timeout_s 900, unit_cells 70, within_cells 180, wording_cells 272) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

comply · no model in the path — a baseline, not a peer column — 2 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 2 runs measured the same system — they differ on floor, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

comply · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 64 chips that all say so.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 63 chips that all say so.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-run the free arms on every change to the corpus, the engine or the answer key -- all of it costs $0.00 and takes under a second. Re-run the key gate before any paid call. Re-buy the paid arm only when the corpus has been made harder: on today's set it is at 299 of 300 with a free arm at 300 of 300, so another paid run would measure nothing new.

In one lineNo framework, deliberately

No framework, and the kit is the argument for why. Every piece a framework would sell here is either eight lines of standard library or is the thing being measured. The whole kit imports nothing outside the standard library, so pip install -r requirements.txt installs nothing and a cold clone runs every free arm immediately.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

segment -> select (Commercial Position withheld) -> prompt -> ONE model call -> parse -> derive (units + envelope, numbers off the page) -> guards -> answer; scoring is a separate pass over the committed result file

The other sideWhat a framework costs you

• You write the retry, the timeout and the parse yourself -- about 300 lines across src/adapters/ and src/prompt.py. That is the price of having no dependency to pin and nothing to break on a version bump.
• There is no streaming, so a long reasoning budget is a silent socket for up to 900 seconds rather than a progress bar. p95 here is 68.3 s.
• There is no provider-side structured output, so a malformed reply is a failure this kit records rather than a schema the provider enforces. 0 of 60 failed here.
• Nothing is shared between calls -- no cache of the fixed system block is negotiated, and the harness records no prompt-cache field, so 76.3 pct of every prompt is re-sent and priced at full rate on every projection row.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-spec-capability on the fast tier, 2026-08-31. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 2 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 2 runs on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- and the MIT text is in the repository's LICENSE-PUBLIC file, NOT in its root LICENSE, which governs the repository itself and grants a reader nothing. Everything under data/corpus/ and data/gold.jsonl was generated for this kit by tools/build_corpus.py and is original to it; there is no third-party material here to relicense, and nothing was scraped, fetched, licensed, anonymised or derived from a real document. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe per-property verdict, and the grades it holds on

whether each requested property is answered usefully. Four classes, and the four errors are NOT interchangeable -- WITHIN called OUTSIDE refuses an order nobody audits; OUTSIDE called WITHIN promises a specification the plant cannot hold; a condition collapsed into WITHIN quotes a grade that cannot hold it; NOT_CHARACTERISED called OUTSIDE reports 'we have never measured it' as 'we cannot do it'. A conditional verdict is only correct if the named grade set is right too.

MethodHow the test was run

python3 -m evals.run --run-id <id> [--floor rules|units|--stub]; evals/scoring.py compares the DERIVED answer -- src/answer.py's, computed from the printed page -- against data/gold.jsonl. No model is in the grading path and re-scoring a cached run is free (--resume --rescore).

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl -- 300 property rows derived through src/envelope.py from the facts tools/build_corpus.py injected, never typed by a person. Graded independently by evals/check_labels.py, which re-derives every label from what each document PRINTS through the arms' own parser: KEY CLEAN over 60 reviews and 300 properties, re-run this turn in 0.04 s. The corpus rebuilds byte-identically at four PYTHONHASHSEED values. Every customer, product family, grade, capability band and technical note is invented -- see data/SOURCES.md. ⚠︎ THE KEY IS DERIVED THROUGH THE SAME ENGINE THE ARMS ARE GRADED THROUGH, so a defect in the containment rule would be invisible to every arm at once; check_labels.py re-derives from the printed page but does not re-implement the rule.

No true/false rates for this grader. It records 7 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold and no sweep: every grader here is exact match against a closed four-word vocabulary, with the named grade set compared as a set where the verdict is conditional. There is no tolerance band, no similarity score and nothing tuned, so there is no number that could have been chosen to flatter a result.

Cadence: Every change to the corpus, the engine or the answer key -- it costs $0.00 and runs in a tenth of a second on all four arms. Re-buy the paid arm only after the corpus has been made harder.

The decisionWhen to reach for it

In one lineThe whole-enquiry answer, its reason and its binding grade

whether the enquiry can be answered at all, on its OWN denominator of 60. Deliberately not nested inside the property score: two properties can each be achievable on grades that share nothing, which is a GRADE CONFLICT and is invisible in any per-property table. Scored with the reason and the binding grade beside it, because CANNOT_MEET for the wrong reason sends the enquiry back asking for the wrong thing.

MethodHow the test was run

Same run, same scorer; src/envelope.py::overall applies the precedence to the derived rows and never reads the arm's stated answer.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The same data/gold.jsonl. Each review's overall answer, reason and binding grade are derived by src/envelope.py::overall from the key's own property rows before any arm answers, and the 30/12/10/8 split across CANNOT_MEET / NOT_ANSWERABLE / CAN_MEET_ON_CONDITION / CAN_MEET is fixed by the corpus.

No true/false rates for this grader. It records 8 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold. Exact match on a closed four-word vocabulary plus the reason code and, where the answer is conditional, the binding grade as a set. Nothing tuned, nothing swept.

Cadence: Same run, same command, same cost -- $0.00.

The decisionWhen to reach for it

In one lineThree checks that need no answer key

whether a reading can be convicted on the document alone. unlocatable_id -- the reply cites an entry, grade or wording NOT printed on the sheet. transcription_mismatch -- the arm's copy of a printed number disagrees with the page. verdict_inconsistent -- the arm's STATED verdict disagrees with what its own matching implies, re-run in code. This is what a forker gets on day one, on enquiries nobody has ever scored.

MethodHow the test was run

Computed inside src/review.py on every paid or stub call and written to the result file; no answer key is read.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: NO ANSWER KEY AT ALL, which is the point. All three checks compute against the DOCUMENT: the set of identifiers and wordings the review prints (src/sheet.py::ids), the printed numbers themselves, and the arm's own matching re-run through src/envelope.py. They therefore return the same reading on an enquiry nobody has ever scored, which is every enquiry a forker owns.

No true/false rates for this grader. It records 5 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold: each check is an exact set or string comparison against what the document prints. Nothing is tuned and nothing is scored on a similarity.

Cadence: Every paid call, automatically, and on every stub run for free.

The decisionWhen to reach for it

In one lineThe answer key, graded before anything is bought

whether the labels themselves are derivable from what each document prints. It re-reads all 60 reviews through the arms' own parser and re-derives every verdict, condition, binding grade, reason and overall answer.

MethodHow the test was run

python3 -m evals.check_labels -- 0.04 s, no key, no network. Re-run this turn.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The printed documents. check_labels.py re-reads all 60 reviews and re-derives every verdict, condition, binding grade, reason and overall answer from what the page states, then compares against data/gold.jsonl. Nothing in the key was typed by a person; tools/build_corpus.py computes it from the facts it injected and refuses to write a unit case unless the unit-blind comparison genuinely lands wrong, or an alias case unless exact-name lookup genuinely misses.

No true/false rates for this grader. It records 4 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold. Byte equality on the rebuild and exact equality on every label; nothing is tuned and nothing is scored on a tolerance.

Cadence: Before every paid run, and after any change to tools/build_corpus.py, src/sheet.py or src/envelope.py.

The decisionWhen to reach for it