The business caseThe problem this solves
A subcontractor sends a submittal package — product data, samples, certifications, test reports, shop drawings, a warranty form. The specification paragraph that asked for each one states particulars: three samples, twelve by twelve inches, a named test method, an independent testing agency, a scale, a term. Today a project engineer reads the section and the package side by side before routing it, and the package that gets through is rarely the one with an obvious hole in it. It is the one where a document of exactly the right type is enclosed, filed under the right heading, citing the right paragraph — and answering a different requirement. the paragraph-by-paragraph read a project engineer does against the specification section before routing a submittal package for review
Audience
The project engineer or submittal coordinator who routes the package, and the subcontractor who will be told to resubmit. The decision is whether a model call per package is worth buying over the checklist you would write instead. Every number on these pages came from one real run of this code, not from a vendor page.
The inputThe actual pre-check packets
The corpus is 40 pre-check packets, 0.20 MB (json 4 · jsonl 1 · md 2 · txt 40). ⚠︎ EVERY BYTE OF IT IS INVENTED, AND THAT IS STATED BEFORE ANY NUMBER. The Kelwin Bay Transit Hall is a fictional project and the Kelwin Bay Project Specification, Issue 3 is a fictional specification written for this kit. Its section numbers, paragraph numbers, test method designations and material standards are made up, and the numbering shape was chosen deliberately so that it does not resemble a real one: it is not MasterFormat, not CSI, not ASTM, not UL and not any real specification or standard. It was generated rather than found because the measurement needs an answer key at the level of one paragraph, and because the corpus has to be SHAPED against the shortcuts: 169 of 426 package items cite the paragraph they are filed against and an UNRESPONSIVE item cites it as often as a responsive one; 70 items are the right document type answering a different requirement, each naming its own type in the same words the paragraph uses; 17 items name a document and enclose nothing; 43 responsive items are reworded end to end with every particular restated equivalently, so that 'is every required phrase present verbatim?' cannot separate responsive from incomplete.
The corpus
- The 40 pre-check packetsgenerated from a fixed seed, so no real record, person or institution appears in it.
- Where each came fromwritten for this kit rather than collected — the corpus is generated in the kit's own repository, so there is no third-party data in it.
Swap this folder for your own material and the kit is pointed at your pre-check packets. That is the whole change — there is no database to migrate.
SUBMITTAL PRE-CHECK PACKET — PKG-0001
Project: Kelwin Bay Transit Hall (fictional) · Transmittal SUB-01113
Subcontractor: Braithe Panel Systems (fictional)
Specification: Kelwin Bay Project Specification, Issue 3 (a fictional specification written for this kit)
PART A — WHAT THE SPECIFICATION REQUIRES TO BE SUBMITTED
Each paragraph below is a submittal requirement. A package answers a paragraph
only if something actually enclosed responds to THAT paragraph, with every
particular it states.
SP-0721-1.04.A.3 Section SP-0721 Thermal Insulation — Product Data
Submit the manufacturer's printed product data stating the thermal resistance per inch, the
density and the water vapour permeance, for each insulation type and thickness scheduled.
SP-0742-1.06.B.1 Section SP-0742 Metal Wall Panels — Shop Drawings
Submit shop drawings at not less than 1 1/2 inches to the foot, showing the panel layout, the
joint locations, the flashing, the closures and the attachment to the substrate, for the whole
of the panelled elevation.
SP-0812-1.04.A.2 Section SP-0812 Interior Door Hardware — Hardware Schedule
Submit a hardware schedule listing every opening, its hardware set, the manufacturer, the
product number, the finish and the handing, in opening number order, and cross-referenced to
the door schedule on the drawings.
SP-0812-1.04.C.1 Section SP-0812 Interior Door Hardware — Sample Warranty
Submit a sample warranty for ten (10) years, covering every closer and exit device against
failure in normal use, on the manufacturer's own printed form, unexecuted.
SP-0951-1.05.B.2 Section SP-0951 Acoustical Ceiling Panels — SamplesAbridged — the file continues.
The outcomeWhat a good result looks like
Every specification paragraph gets one of four verdicts and, where the package does answer it, the item and the words it rests on — so the coordinator reads a difference rather than a claim about one.
And when it cannot
It made 3 errors in 320. Two are enclosures the key calls UNRESPONSIVE and it graded differently — one as INCOMPLETE, one as MISSING — and one is the mirror image, an INCOMPLETE certificate it called UNRESPONSIVE. Every one of the three was still reported as a deficiency. It waved nothing through: 0 of the 211 paragraphs the key calls deficient were reported RESPONSIVE, against the tuned free floor's 72.
Where it fitsWhat did work
Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.
- you need a checklist of documents nobody enclosed at all — the free
bestfloor — 0 calls, $0.00
it scores 314 of 320 on presence against the paid call's 319. Six items to one, exact two-sided p = 0.125000: on this corpus there is NO MEASURABLE DIFFERENCE, and the code costs nothing and reaches no network - you need to know whether what WAS enclosed actually answers the paragraph — the paid call
253 of 256 against the floor's 193, p < 0.000001, and it waved through 0 deficiencies against the floor's 72 - a first pass over a backlog where a coordinator reads every flag — the paid call, at $0.000911 a package
it over-flags rather than under-flags: 0 of the 211 deficient paragraphs were reported as responsive, and every one of its 3 errors still reached the coordinator as a deficiency
And where nothing here is good enough:
- you want a number you can quote about your own specification — neither, until you re-run the eval on your own labelled packages
the Kelwin Bay Project Specification is invented and its shortfalls come from a published list of weaker values. Real packages fall short in ways nobody enumerated
At a glanceHow the whole thing runs
Run once, for real, on 2026-09-11. Every figure on these pages was captured from that run — nothing is written from intent.
14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.
Should you use this?What you bring, where it stops, and when not to use it
Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.
| What do I have to bring? | Replace data/spec.json with your own section — paragraph ids, document types, requirement text and the particulars you consider material — and drop your packages into data/corpus/ in the two-part shape tools/build_corpus.py emits. THE MEASURED ACCURACY DOES NOT TRAVEL WITH YOUR CORPUS. Corpus lens → |
| When is this the wrong choice? | Avoid: Quoting its four-way verdict — it waves 72 deficiencies through as responsive and finds only 11 of the 70 unresponsive enclosures. That is the case against the best-fitting scenario (“you need a checklist of documents nobody enclosed at all”). 4 scenarios scored in all, each with its own. Eval lens → |
| Where does it stop working? | a package that answers a paragraph BY REFERENCE to a document not in the transmittal — the correct answer is then 'cannot be determined from this package' and the schema has no verdict for it. 5 recorded failure modes, each from a run rather than a guess. Corpus lens → |
| What was never verified? | whether any of these rates survives on a REAL specification section. The Kelwin Bay Project Specification is invented, its paragraphs are short and self-contained, and its unresponsive enclosures were written to be plausible — a real section cross-refers, is amended by addenda, and receives substitution requests. 5 items this kit says it could not check. Eval lens → |
| Can I run this on a model I control? | The shipped adapter is the runtime provider is not named on this page; the Prompt lens states what swapping it costs. The published figures come from 1 model on the fast tier. Prompt lens → |
| And if it fits — what do I stand up? | 5 artifacts with a stated home and a stated egress, and 4 decisions each with what you provision past its ceiling — plus what was not measured. That is the next page, not this one. step 14 — Run it in your environment → |
Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).
Last verified 2026-09-11 — r001-submittal-precheck. Every figure on these pages was captured from that run.
Run itHow this reaches your data
Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.
Run this on your own data
- The pipeline, its eval harness and the runs behind every numberdeployed inside your environment, on your own model endpoints, against your own documents.
- The corpus above is the shape, not the limitit is a folder swap, and there is no database to migrate.
Checked before this shipped — A clean checkout with no key configured renders the whole board — the corpus, the answer key, the committed run and all four free floors — and scores every floor offline in pure Python. Shot as kits/UC0389-submittal-precheck/docs/shots/submittal-precheck-empty.png against a server started with API_KEY blanked in its own environment.
