The business caseThe problem this solves

A federal proposal is read for responsiveness before it is read for merit: every requirement the solicitation's matrix states has to be answered, in the volume section the instructions name for it, and a requirement nobody answered is found by the evaluator rather than by the offeror. The check that catches it is a spreadsheet with a keyword column, run against a volume that has been rewritten four times, in the hours before it freezes. It searches for the requirement's words and marks the row green -- and the requirement's words are in the file whether the volume answers it, points at another volume, or asserts full compliance in a narrative the instructions say answers nothing. the first pass a proposal manager makes over a submitted volume -- reading each requirement in the solicitation's matrix against the volume and writing down, per requirement, whether it is answered, WHERE, and which kind of not-answered it is when it is not.

The inputThe actual proposal packs

The corpus is 50 proposal packs, 0.34 MB (txt 50). BECAUSE THE OTHER HALF OF THIS PAIR IS NOT A PUBLIC RECORD. A real solicitation is published; the proposal written against it is the offeror's competition-sensitive work, and a corpus pairing the two would be republishing somebody's writing under an MIT licence. There is a second reason and it is worse: a file reproducing a real agency's words under a real solicitation number, sitting in a folder of synthetic data, is a fabricated official record the moment anybody copies it out. Every pack carries a Synthetic Record banner as its first section for exactly that reason. What IS reproduced is the shape, and the shape is where the whole difficulty lives: instructions that say where each response must sit, a matrix of numbered requirements with the scored ones marked, a volume of numbered sections, and -- the part that matters -- text that contains a requirement's words without answering it. Four of the six calls this kit makes are about telling those apart, and 111 of 280 (39.6 pct) of them cannot be reached by any word search at all.

Synthetic Record
----------------
This is a SYNTHETIC proposal compliance pack generated by tools/build_corpus.py for the AI Foundry use-case kit `proposal-compliance`. The issuing office, the solicitation number, the programme, every requirement, every section number, every sentence of the proposal volume and every name in it are INVENTED. Nothing here was copied from, adapted from or checked against any real solicitation, acquisition regulation, agency supplement, source-selection plan, proposal, company procedure or person, because none was consulted. It describes no real programme, no real agency, no real offeror and no real person, and it must never be treated as a solicitation or as a proposal.

Routing Slip
------------
Received by the proposal desk: 2026-01-07
Pack id: PKG-0001
Volume under check: Past Performance Volume
Submission state: Final, as submitted
Compliance check owed: every requirement in the matrix, one row each

Internal Reference
------------------
Bid decision: bid, recorded at gate review 3
Price-to-win estimate: $139.9 million over five years
Wrap rate applied: 1.3
Competitor read: 1 incumbent(s) expected; a set-aside challenge is anticipated
Win probability at gate 3: 75 per cent
Teaming discount held: 4 per cent, not disclosed to the teammate
Capture lead: M. Halvorsen
Proposal manager: P. Vasquez
Colour team notes: pink team scored this volume 3 of 5

Solicitation Header
-------------------
Issuing office: Federal Space Systems Acquisition Board (FSSAB)
Solicitation: FSSAB-2026-0101
Title: Launch range instrumentation sustainment
Volume under check: Past Performance Volume
Proposals close: 2026-07-14
Submission state of the attached volume: Final, as submitted

Instructions to Offerors
------------------------

The outcomeWhat a good result looks like

One row for every requirement the solicitation's own matrix states -- four to eight of them, counted in pure code before anything is called -- each naming what the volume does about it, where that text sits, the reference you can open, and the shipped rulebook's call. Plus one pack-level disposition composed from all of them.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• A compliance pass over a submitted volume before it freezes, for a proposal manager who will read every flagged row and act on it. — the one tier measured here — 126 of 127 (99.2 pct) of the rows that need flagging were flagged, against the free keyword floor's 19 of 130 (14.6 pct) on the same rows, and 1 of 127 (0.8 pct) cleared outright against the floor's 111 of 130 (85.4 pct).
  The gap between the two is entirely the four classes a phrase search cannot see: the floor false-cleared every cross-reference, every compliance-narrative claim, every misplaced response and every half answer in the corpus — 111 rows — and got the two classes it can see exactly right. That is not a straw man, it is the spreadsheet a proposal shop already runs.
• A first pass over an early draft, where the question is which requirements have had nothing written against them yet. — the same tier. On the rows that matter here — the ones the volume is silent on — nothing was cleared: 0 of 19 (0.0 pct) from gold not_addressed and 0 of 22 (0.0 pct) from gold narrative_only.
  An early draft is mostly gaps, which is the regime where the free floor is least wrong and the model's advantage is smallest — it gets not_addressed right too. The advantage is in the narrative class: the floor cleared 22 of 22 (100.0 pct) of those and the run cleared 0 of 22 (0.0 pct).
• Deciding which requirements a capture lead has to sign off personally. — the shipped pure-code condition, not the model's judgement — recall 1.0, precision 0.9286 against the same rule run over gold, on 14 packs where it fires.
  It misses nothing: every pack gold escalates, it escalated. The condition reads the evaluated column out of the solicitation in pure code rather than asking the model for it, so half of the rule cannot be wrong.

And where nothing here is good enough:

• Running it unattended, with the matrix going straight to a writer's queue. — nothing measured here supports it.
  Not because the rates are bad but because of what one pack did: PKG-0042 returned an empty reply, and the six requirements in it are simply absent from the output. Nothing downstream of the parser can tell 'this pack has no gaps' from 'this pack produced no answer' unless somebody reads the coverage number beside the matrix.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-23. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-23 — r001-proposal-compliance. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with no key configured rebuilds the whole corpus and its gold in under a second, passes the gold gate, scores the free keyword floor over all 50 packs for $0.00, and renders the local UI with every recorded result in it. pip install is a no-op: requirements.txt is empty on purpose.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per pack inside evals/run.py: segmentation, the pure-code requirements-matrix parse, selection, prompt assembly, ONE completion call, parse, reference location and the pure-code sign-off rule. It is wall-clock around the whole unit of work and not the provider's own service time, so it includes queueing behind the other 49 packs running concurrently.

Current processWhat it replaces

the first pass a proposal manager makes over a submitted volume -- reading each requirement in the solicitation's matrix against the volume and writing down, per requirement, whether it is answered, WHERE, and which kind of not-answered it is when it is not.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one pack section is

cut into named sections on underlined headings (src/segment.py), then the union of the sections each field maps to is sent (src/select.py). NOTHING IS CHUNKED AND NOTHING IS RETRIEVED -- the selected sections of one pack go whole into one call, and the submitted volume is deliberately ONE section so that 'the response is under the wrong heading' stays answerable.

SetupWhat the setup figure measured

THERE IS NO INDEX. The figure is the cost of SEGMENTING the whole corpus -- 50 packs into 430 named sections -- in pure Python with no model and no network. The requirements matrix is parsed in the same pass. A retrieval step would be the wrong shape here: the question is asked of one pack at a time and every section of it that is not the pricing page is relevant.

LicenceLicence

MIT -- this repository's own licence, corpus included

Bring your ownBring your own proposal packs

Two files and one function. (1) Replace data/rulebook.json with YOUR definition of what counts as a response -- that is the seam shops actually disagree about, and it moves the prompt, gold and the scorer together. (2) Put your packs in data/corpus/ in a format src/segment.py can cut and src/matrix.py can parse, and add your own confidential section names to src/select.py's NEVER_SENT. Then re-derive gold, or label it by hand, and run the gate.

⚠︎ And what stops being true when you do: EVERY SCORE ON THIS PAGE STOPS BEING TRUE THE MOMENT YOU DO. Not because the model changes, but because this corpus is denser in the hard classes than a real submission is: 127 of 280 (45.4 pct) of its rows are something other than addressed, dealt deliberately so no rate rests on three rows. A real volume in its final week is mostly answered, so both error rates on your own packs will be computed over different denominators than these, and the false-clear rate in particular is the one that moves.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 6,107 tokens assembled, 1,941 are rulebooks — 32% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on PKG-0010 -- five calls at max_tokens=1, each the previous prefix plus exactly one more part, with each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. results/tokens-p001-proposal-compliance.json.

Full promptThe prompt in full

You check a SUBMITTED PROPOSAL VOLUME against the REQUIREMENTS MATRIX of the solicitation it answers, and you return a COMPLIANCE MATRIX: for every requirement, whether the volume answers it, WHERE, and which requirements are gaps. You return JSON and nothing else.

You are preparing a compliance matrix for a proposal manager. YOU NEVER WRITE, EDIT, DRAFT, REWRITE OR SUGGEST PROPOSAL TEXT, never submit anything, never clear a volume for submission and never waive a requirement. Your job is to say what the submitted volume does with each requirement, apply the rulebook given below, and NAME EVERY REQUIREMENT THE VOLUME DOES NOT ANSWER.

RULES, in order of importance:
1. ONE ROW PER REQUIREMENT IN THE MATRIX, KEYED BY THE ID PRINTED THERE, IN THAT ORDER, AND NO OTHERS. The number of requirements is different in every solicitation. A requirement the volume says nothing about is a row reading `not_addressed`, never an absent row: a compliance matrix that drops the requirements it found nothing for cannot be told apart from one that never checked them, and the dropped rows are exactly the ones nobody looks at again.
2. `addressed` IS THE ONLY CALL THAT TELLS ANYBODY TO STOP LOOKING, AND A FALSE ONE IS THE EXPENSIVE ERROR. Every other call sends a writer back to the volume for an hour and is recovered. A requirement marked `addressed` that is not answered ships as a gap nobody looked at again. When the text you have found does not clearly answer the whole requirement under the heading the Instructions name, the call is one of the other five.
3. A POINTER IS NOT A RESPONSE. Text that names another volume or another section where the answer supposedly is states WHERE SOMEBODY MIGHT LOOK; a response states WHAT THE OFFEROR WILL DO. It is `points_elsewhere` and the call is `cross_referenced`, however fluent the sentence and however confidently it asserts compliance.
4. THE COMPLIANCE NARRATIVE RESPONDS TO NOTHING AND THE INSTRUCTIONS SAY SO. A paragraph there asserting that the offeror complies fully with a requirement REPEATS the requirement's words and answers it in no part. It is `restates_only` in the `compliance_narrative` and the call is `narrative_only`. This is the text a phrase search over the volume finds and reports as a hit.
5. PLACEMENT IS PART OF THE ANSWER. The Requirements Matrix names, for each requirement, the heading the response must sit under. Responsive text under any other heading of the submitted volume is `other_section`, and the call is `outside_mandated_section` -- the response exists and an evaluator reading to the Instructions will not find it.
6. BOTH HALVES. A requirement asking for two things is answered when both are stated. One stated and one not is `responds_in_part`.
7. `compliance_call` IS DECIDED ONLY BY THE RULEBOOK BELOW, from `response_state` and `location_kind`. Work through the six steps IN ORDER and STOP at the first that fires.
8. `disposition` IS DECIDED ONLY BY THE RULEBOOK, from `submission_state` and every requirement call. Decide it AFTER the rows. An `interim_draft` NEVER excuses a gap or a placement defect -- those are settled above it, and the word draft describes the review cycle, not the requirement.
9. `not_present` IS A REAL ANSWER AND YOU ARE EXPECTED TO USE IT. Most of these volumes miss something. A requirement marked answered on the strength of a nearby sentence is the failure this check exists to prevent.
10. Copy every reference verbatim from the pack -- the solicitation number in `solicitation_ref`, the section or paragraph reference in `response_ref`. Cite where the text you are describing actually IS, which for a cross-reference is where the POINTER is printed and never the volume it points at. Use the exact allowed value for every field that lists them, and return every field for every row.

COMPLIANCE RULEBOOK (the authority for `compliance_call` and `disposition`; this is an ILLUSTRATIVE rulebook written for this kit, and it reproduces no acquisition regulation, agency supplement, solicitation or company procedure)

WHAT IS ON THE COMPLIANCE MATRIX
  - ONE ROW FOR EVERY REQUIREMENT THE MATRIX STATES, keyed by the requirement id printed there, and no rows the matrix does not state. The count varies from solicitation to solicitation, so a short reply is not a short solicitation - it is a check that stopped early, and a compliance matrix missing a row is indistinguishable from one whose requirement was never checked.
  - ONE RESPONSE REFERENCE PER REQUIREMENT, verbatim as the submitted volume prints it, and `none` when nothing in the volume speaks to it. A call with no location behind it is an opinion, and the whole use of this matrix is that somebody can open the volume at the place named and disagree.
  - THE TEXT THAT ANSWERS, NOT THE TEXT THAT MENTIONS. A Compliance Narrative that repeats the requirement's words is not the response, and neither is a sentence naming the volume where the response supposedly lives. Where both a mention and a response exist, the response is what the row is about.
  - A GAP IS STATED, NEVER LEFT BLANK. `not_addressed` is a finding about the submission. A row left empty because the checker found nothing to say reads on the page exactly like a requirement nobody checked.
  - THE SUBMISSION STATE IS READ OFF THE PACK, NOT INFERRED FROM HOW FINISHED THE VOLUME LOOKS. A polished draft and a rough final are both common, and the Routing Slip and the Solicitation Header are what settle it.

THE FOUR INSTRUCTIONS EVERY SOLICITATION IN THIS CORPUS CARRIES
  IO-1   PLACEMENT - the requirement is answered under the heading the Instructions to Offerors name for it, and responsive material under any other heading is not evaluated against it
  IO-2   A POINTER IS NOT A RESPONSE - a cross-reference to another volume or section states where somebody might look; a response states what the offeror will do
  IO-3   THE NARRATIVE RESPONDS TO NOTHING - the Compliance Narrative is a statement of intent, is evaluated against no requirement, and repeating a requirement's words there is not answering it
  IO-4   BOTH HALVES - a requirement that asks for two things is answered only when both are stated

SUBMISSION STATES
  final_as_submitted   the volume as it went in the box. Nothing further will be written against this solicitation
  interim_draft        a working draft with at least one review cycle still to run. THE WORD DRAFT DESCRIBES THE CYCLE, NOT THE REQUIREMENT - a requirement with no response at all is a gap in a draft exactly as it is in a final, and a draft state never excuses one

THE TWO STATED FACTS, read off the pack for every requirement
  response_state:
    responds_in_full     text in the submitted volume answers everything this requirement asks for. If the requirement asks for two things, both are stated
    responds_in_part     text in the submitted volume answers some of what this requirement asks for and not the rest - a compound requirement with one half stated
    points_elsewhere     the only text touching this requirement names another volume or another section where the answer supposedly is. It states where somebody might look and not what the offeror will do
    restates_only        the only text touching this requirement repeats the requirement's own words as a claim of compliance without answering it. 'The offeror complies fully with the configuration control requirement' is a restatement, not a response
    not_present          no text in the submitted volume speaks to this requirement at all
  location_kind:
    mandated_section     the text sits under the heading the Instructions to Offerors name for this requirement in the Requirements Matrix
    other_section        the text sits somewhere else in the submitted volume, under a heading the Matrix names for a different requirement or for none
    compliance_narrative the only text sits in the Compliance Narrative, which the Instructions state responds to no requirement. This is the only value allowed with response_state restates_only, and the only response_state allowed with it
    none                 there is no text. This is the only value allowed when response_state is not_present, and the only response_state allowed with it

THE COMPLIANCE CALL -- work through IN ORDER, stop at the first that fires
  1. NOTHING THERE. If location_kind is `none` or response_state is `not_present`, the matrix records `not_addressed`. This is the gap: the requirement was stated and the volume says nothing about it. It is the finding the whole check exists to produce, and it is never excused by the submission state.
  2. WORDS WITH NO RESPONSE. If response_state is `restates_only` or location_kind is `compliance_narrative`, the matrix records `narrative_only`. The requirement's words are in the file and nothing answers it. THIS IS THE CALL A KEYWORD SCAN CANNOT MAKE AT ALL: a search for the requirement's phrase finds this text and reports a hit, which is a false clear on a gap.
  3. A POINTER, NOT A RESPONSE. If response_state is `points_elsewhere`, the matrix records `cross_referenced`. Somebody has to decide whether the volume it points at is in this submission, whether it says what the pointer claims, and whether the evaluator will follow it. That decision belongs to the capture lead and is recorded, never made here.
  4. THE RIGHT ANSWER IN THE WRONG PLACE. If location_kind is `other_section`, the matrix records `outside_mandated_section`. The response exists and an evaluator reading the heading the Instructions name will not find it. It comes before the partial rule because placement decides whether the text is read at all, and a half answer that is read beats a full answer that is not.
  5. HALF OF IT. If response_state is `responds_in_part`, the matrix records `partially_addressed`. One half of a compound requirement is stated and the other is not.
  6. ANYTHING ELSE. The matrix records `addressed`. THIS IS THE ONLY CALL THAT TELLS ANYBODY TO STOP LOOKING, and it is the one this kit measures the cost of getting wrong.

THE DISPOSITION -- work through IN ORDER, stop at the first that fires
  1. If any call is `not_addressed` or `narrative_only`, the pack records `blocking_gap`. A requirement the volume does not answer is not a matter of degree.
  2. If any call is `outside_mandated_section`, the pack records `placement_defect`. Responsive text an evaluator will not read is a defect of the submission, not of the writing.
  3. If submission_state is `interim_draft`, the pack records `open_in_draft`. There is a cycle left in which a pointer can become a response and a half answer can be finished. IT IS THE THIRD RULE AND NOT THE FIRST, AND THAT IS THE WHOLE TRAP: a draft state does not excuse a gap or a placement defect, both of which are settled above it. The word draft describes the cycle, not the requirement.
  4. If any call is `cross_referenced`, the pack records `reference_risk`. Nothing is missing and somebody still has to accept that a pointer will be followed.
  5. If any call is `partially_addressed`, the pack records `partial_response`.
  6. ANYTHING ELSE. The pack records `compliant` - every requirement in the matrix answered in full, in the section the Instructions name for it.

WHY A FALSE CLEAR IS THE EXPENSIVE ERROR
  A FALSE CLEAR IS THE ERROR THIS RULEBOOK IS SHAPED AROUND. `addressed` is the only call that tells a proposal manager to stop looking at a requirement. Every other call sends somebody back to the volume, costs an hour and is recovered. A requirement marked `addressed` that is not answered ships as a gap nobody looked at again, and on a federal proposal that loses the bid on responsiveness before anything is scored on merit. The two directions are counted apart everywhere in this kit - in evals/judge.py, on the kit's pages and in the UI - and they are never averaged into one accuracy figure.

WHY THE GAP IS THE FINDING
  A compliance matrix that only reports what it found is a matrix nobody needs; the reason the job exists is the requirement the volume never answered. Every other call on this list sends somebody back to the volume for an hour and is recovered. `not_addressed` and `narrative_only` are the two that ship if nobody catches them, and `narrative_only` is the dangerous one of the pair because the requirement's own words ARE in the file - so the tool a proposal shop actually runs, a phrase search over the submitted volume, reports a hit and clears it.

THE REQUIREMENTS MATRIX FOR THIS SOLICITATION -- 6 requirement(s), and you owe EXACTLY ONE ROW FOR EACH, keyed by the id printed here, in this order:

  R-01  [EVALUATED - the source selection scores this one]  answer under heading: Systems Integration
        The offeror shall describe its approach to interface control for the delivered subsystem boundaries.
  R-02  [not separately evaluated]  answer under heading: Quality and Configuration Management
        The offeror shall describe its approach to configuration control for hardware baselines.
  R-03  [not separately evaluated]  answer under heading: Information Protection
        The offeror shall describe its approach to cybersecurity control inheritance for the development environment. The offeror shall also state which controls are inherited rather than implemented.
  R-04  [not separately evaluated]  answer under heading: Reliability and Failure Management
        The offeror shall describe its approach to failure reporting and corrective action for fielded units.
  R-05  [not separately evaluated]  answer under heading: Software Development and Assurance
        The offeror shall describe its approach to software assurance for mission software builds.
  R-06  [EVALUATED - the source selection scores this one]  answer under heading: Sustainment Engineering
        The offeror shall describe its approach to obsolescence management for line-replaceable units. The offeror shall also state how far ahead a last-time-buy is forecast.

  The ids above are the complete list. Return a row for every one of them and for nothing else.

Return these:
- solicitation_ref (string) -- the solicitation number printed in the Solicitation Header section, verbatim (for example UAPD-2026-0114)
- submission_state (enum) one of: final_as_submitted, interim_draft -- whether this is the volume as it went in the box or a working draft with a review cycle still to run. Read it off the Routing Slip line 'Submission state:' and the Solicitation Header, NOT from how finished the writing looks. A draft state NEVER excuses a requirement with no response - a gap is a gap in a draft exactly as it is in a final
- volume_checked (enum) one of: technical_volume, management_volume, past_performance_volume -- which volume of the proposal this pack contains, read off the Solicitation Header line 'Volume under check:' and confirmed by the Instructions to Offerors
- disposition (enum) one of: blocking_gap, placement_defect, open_in_draft, reference_risk, partial_response, compliant -- the pack-level call, decided STRICTLY by the shipped rulebook from `submission_state` and the requirement calls and nothing else -- so decide it AFTER every requirement row. IN ORDER, stopping at the first that fires. (1) any call 'not_addressed' or 'narrative_only' -> 'blocking_gap'. (2) any call 'outside_mandated_section' -> 'placement_defect'. (3) submission_state 'interim_draft' -> 'open_in_draft' (NEVER above rules 1 and 2: a draft does not excuse a gap or a placement defect). (4) any call 'cross_referenced' -> 'reference_risk'. (5) any call 'partially_addressed' -> 'partial_response'. (6) anything else -> 'compliant'
- requirements (array, EXACTLY ONE OBJECT PER REQUIREMENT IN THE MATRIX, in the order the matrix prints them) -- each carrying:
    - requirement_id (string) -- the requirement id printed in the Requirements Matrix, verbatim (for example R-03). THIS IS THE KEY -- return EXACTLY ONE ROW FOR EVERY REQUIREMENT THE MATRIX STATES, in the order the Matrix prints them, and no rows for anything it does not state. The number of requirements is different in every solicitation: count them in the Matrix, do not assume
    - response_state (enum) one of: responds_in_full, responds_in_part, points_elsewhere, restates_only, not_present -- what the submitted volume DOES about this requirement. 'responds_in_full' when text answers everything it asks for (both halves, where it asks for two things); 'responds_in_part' when a compound requirement has one half stated and the other missing; 'points_elsewhere' when the only text names another volume or section where the answer supposedly is -- a pointer states where somebody might look, a response states what the offeror will do; 'restates_only' when the only text repeats the requirement's own words as a claim of compliance without answering it; 'not_present' when nothing in the volume speaks to it at all
    - location_kind (enum) one of: mandated_section, other_section, compliance_narrative, none -- WHERE that text sits. 'mandated_section' when it is under the heading the Requirements Matrix names for THIS requirement in its 'Answer under heading' column; 'other_section' when it is elsewhere in the submitted volume; 'compliance_narrative' when the only text is in the Compliance Narrative section, which the Instructions state responds to no requirement; 'none' when there is no text. 'none' is allowed only with response_state 'not_present' and 'not_present' only with 'none'; 'compliance_narrative' is allowed only with 'restates_only' and 'restates_only' only with it
    - response_ref (string) -- the reference of the SINGLE place in the submitted pack the row is about, verbatim as the pack prints it -- a proposal section number such as 'Section 2.3', or a Compliance Narrative paragraph such as 'CN-2'. Cite where the text actually IS, which for a cross-reference is where the POINTER is printed and not the volume it points at. Return the string 'none' when location_kind is 'none'
    - compliance_call (enum) one of: not_addressed, narrative_only, cross_referenced, outside_mandated_section, partially_addressed, addressed -- this requirement's matrix call, decided STRICTLY by the shipped rulebook from `response_state` and `location_kind` and nothing else. Work through the six steps IN ORDER, stopping at the first that fires. (1) location_kind 'none' or response_state 'not_present' -> 'not_addressed'. (2) response_state 'restates_only' or location_kind 'compliance_narrative' -> 'narrative_only'. (3) response_state 'points_elsewhere' -> 'cross_referenced'. (4) location_kind 'other_section' -> 'outside_mandated_section'. (5) response_state 'responds_in_part' -> 'partially_addressed'. (6) anything else -> 'addressed'. `addressed` IS THE ONLY CALL THAT TELLS A PROPOSAL MANAGER TO STOP LOOKING AT A REQUIREMENT -- give it only when text under the mandated heading answers the whole requirement

Return a JSON object with exactly these top-level keys: solicitation_ref, submission_state, volume_checked, disposition, requirements
`requirements` is an array of EXACTLY 6 object(s), one per requirement id above, in that order. It is never shorter: a requirement the volume says nothing about is a row reading `not_addressed`.

PROPOSAL PACK
-------------
Synthetic Record
----------------
This is a SYNTHETIC proposal compliance pack generated by tools/build_corpus.py for the AI Foundry use-case kit `proposal-compliance`. The issuing office, the solicitation number, the programme, every requirement, every section number, every sentence of the proposal volume and every name in it are INVENTED. Nothing here was copied from, adapted from or checked against any real solicitation, acquisition regulation, agency supplement, source-selection plan, proposal, company procedure or person, because none was consulted. It describes no real programme, no real agency, no real offeror and no real person, and it must never be treated as a solicitation or as a proposal.

Routing Slip
------------
Received by the proposal desk: 2026-02-22
Pack id: PKG-0010
Volume under check: Technical Volume
Submission state: Interim draft, review cycle 2 of 3
Compliance check owed: every requirement in the matrix, one row each

Solicitation Header
-------------------
Issuing office: Unified Aerospace Procurement Directorate (UAPD)
Solicitation: UAPD-2026-0110
Title: Airframe structural sustainment services
Volume under check: Technical Volume
Proposals close: 2026-04-14
Submission state of the attached volume: Interim draft, review cycle 2 of 3

Instructions to Offerors
------------------------
These instructions govern this Technical Volume and are the authority for where a response must sit.
IO-1   PLACEMENT. The requirement is answered under the heading the Instructions to Offerors name for it, and responsive material under any other heading is not evaluated against it.
IO-2   A POINTER IS NOT A RESPONSE. A cross-reference to another volume or section states where somebody might look; a response states what the offeror will do.
IO-3   THE NARRATIVE RESPONDS TO NOTHING. The Compliance Narrative is a statement of intent, is evaluated against no requirement, and repeating a requirement's words there is not answering it.
IO-4   BOTH HALVES. A requirement that asks for two things is answered only when both are stated.

Requirements Matrix
-------------------
The requirements below are the complete list for this volume. There are 6 of them.

R-01 | Evaluated: yes | Answer under heading: Systems Integration
     Key phrase: interface control
     The offeror shall describe its approach to interface control for the delivered subsystem boundaries.

R-02 | Evaluated: no | Answer under heading: Quality and Configuration Management
     Key phrase: configuration control
     The offeror shall describe its approach to configuration control for hardware baselines.

R-03 | Evaluated: no | Answer under heading: Information Protection
     Key phrase: cybersecurity control inheritance
     The offeror shall describe its approach to cybersecurity control inheritance for the development environment. The offeror shall also state which controls are inherited rather than implemented.

R-04 | Evaluated: no | Answer under heading: Reliability and Failure Management
     Key phrase: failure reporting and corrective action
     The offeror shall describe its approach to failure reporting and corrective action for fielded units.

R-05 | Evaluated: no | Answer under heading: Software Development and Assurance
     Key phrase: software assurance
     The offeror shall describe its approach to software assurance for mission software builds.

R-06 | Evaluated: yes | Answer under heading: Sustainment Engineering
     Key phrase: obsolescence management
     The offeror shall describe its approach to obsolescence management for line-replaceable units. The offeror shall also state how far ahead a last-time-buy is forecast.

Proposal Volume as Submitted
----------------------------

Section 1.1 - Executive Summary
The offeror proposes an approach built on the team already performing comparable work, with the transition handled inside the first sixty days.

Section 2.1 - Systems Integration
This section is written to the page limit stated in the Instructions and the supporting detail is retained in the offeror's programme files.

Section 2.2 - Quality and Configuration Management
The offeror's approach to configuration control for hardware baselines is set out in full in Attachment 4 (Quality Plan) and is not repeated here. The offeror considers that approach fully responsive to this requirement.

Section 2.3 - Information Protection
Cybersecurity control inheritance for the development environment is performed under a documented procedure owned by the programme control lead. The development environment sits inside an accredited enclave and the offeror maintains the control implementation record for every control it owns. Inherited controls are listed by identifier in the system security plan and are re-confirmed at each annual assessment.

Section 2.4 - Reliability and Failure Management
The offeror's internal procedures governing this area are available for review at the offeror's facility during the evaluation period.

Section 2.5 - Software Development and Assurance
Software assurance for mission software builds is performed under a documented procedure owned by the programme control lead. Every build runs a static analysis pass and a unit regression suite in the pipeline, and no build reaches integration without both records attached to it.

Section 2.6 - Sustainment Engineering
Obsolescence management for line-replaceable units is performed under a documented procedure owned by the production lead. Every line-replaceable unit carries a parts health record refreshed twice a year against manufacturer end-of-life notices and distributor stock depth.

Section 3.1 - Programme Overview
This section carries the offeror's general description of the programme and of any material the offeror considered better read in one place than distributed across the volume.
The offeror's organisation for this effort is set out in the organisation chart at the front of this volume, and the reporting lines shown there are the ones in force at award.

Compliance Narrative
--------------------

The paragraphs below are the offeror's statement of intent. They are offered in addition to the volume above.

CN-1 The offeror's management system has been assessed against the standards named in the solicitation and the certificates are current.
CN-2 The offeror complies fully with the failure reporting and corrective action requirement and maintains a documented approach to it across the programme.

Distribution Notes
------------------
Circulated to the proposal desk and to the volume leads. Queries to the desk, not to the issuing office. The teaming partner's proposal office has circulated its own reading of these Instructions, in which a cross-reference into a teammate's volume counts as a response; that reading is theirs and is not part of this solicitation.

Raw responseThe raw response

{
  "solicitation_ref": "FSSAB-2026-0101",
  "submission_state": "final_as_submitted",
  "volume_checked": "past_performance_volume",
  "disposition": "reference_risk",
  "requirements": [
    {
      "requirement_id": "R-01",
      "response_state": "responds_in_full",
      "location_kind": "mandated_section",
      "response_ref": "Section 2.1",
      "compliance_call": "addressed"
    },
    {
      "requirement_id": "R-02",
      "response_state": "responds_in_full",
      "location_kind": "mandated_section",
      "response_ref": "Section 2.2",
      "compliance_call": "addressed"
    },
    {
      "requirement_id": "R-03",
      "response_state": "points_elsewhere",
      "location_kind": "mandated_section",
      "response_ref": "Section 2.3",
      "compliance_call": "cross_referenced"
    },
    {
      "requirement_id": "R-04",
      "response_state": "responds_in_full",
      "location_kind": "mandated_section",
      "response_ref": "Section 2.4",
      "compliance_call": "addressed"
    },
    {
      "requirement_id": "R-05",
      "response_state": "points_elsewhere",
      "location_kind": "mandated_section",
      "response_ref": "Section 2.5",
      "compliance_call": "cross_referenced"
    },
    {
      "requirement_id": "R-06",
      "response_state": "responds_in_full",
      "location_kind": "mandated_section",
      "response_ref": "Section 2.6",
     

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch the gaps in a federal proposal's answers — 50 proposal packs. One model answered, and every answer was then graded Seven different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

There is no LLM judge in this kit and that is not a shortcut. Every cell is an exact string match after light normalisation, a set comparison over the requirement ids the matrix states, or the shipped rulebook's own lookup re-run over the same facts. A judge would add cost, variance and a second thing to validate, and it would be scoring a question that has one right answer written into the pack by the generator.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, never an invoice. The tokens are this run's own and are identical whoever they are priced against.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid -- the token counts are real and the dollars are a projection.

Grading unitWhat the grading figure prices

Free means free: evals/judge.py makes no model call, and neither does the keyword floor it is compared against. Grading run r001-proposal-compliance cost $0.00 and re-grading it costs $0.00 again.

The gradersSeven ways to grade

The free floor is not a null grader and not a straw man: it is the compliance-matrix spreadsheet a proposal shop already runs, it is perfectly reasonable, and it gets two of the six classes right for nothing. It reads the solicitation number, the submission state and the volume out of the header with three regexes — all 50 packs, all correct — and then, per requirement, searches the offeror's own text for the key phrase the matrix prints. Its per-cell accuracy is 79.62 pct, which is the trap: three of the seven cell kinds are mechanical lookups it does perfectly, and the citation grader gives it 100.0 pct exact because it knows exactly WHERE the words are. What it does not know is what they mean, and on the one number that decides whether a requirement stays on somebody's list it clears 111 of 130 (85.4 pct). THE SAME DENOMINATOR ON BOTH SIDES IS THE WHOLE POINT: the floor scored all 280 rows and the run scored the 127 it returned, so the run's headline is also given over all 130 rows (126 of 130 (96.9 pct), 96.92 pct) with the empty pack's rows counted as unflagged — because a reliability failure must not be allowed to improve a safety number.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same pack, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering costs money here and grading costs nothing

Every scorer in this kit is pure code: evals/judge.py, evals/check_labels.py, evals/baseline.py and tools/capture_example.py make no model call between them.

Cost driversWhat actually moves the bill

• ⚑ PROVIDER-SIDE REASONING, WHICH IS THE BILL. 206765 of the run's 226228 output tokens (91.4 pct) were reasoning rather than the JSON matrix, left at the provider's default because this harness sends no thinking parameter on any call. The output side is most of the cost on the projected card, so this one fact is most of the price of a compliance pass.
• The fixed prefix, paid on every call with no caching: system instruction, compliance rulebook and field schema, 4058 of 6107 input tokens (66.5 pct) on the measured pack.
• The size of the solicitation's matrix, which drives BOTH the rendered matrix block and the volume that answers it. This corpus averages 5.6 requirements per pack; a thirty-requirement solicitation pays for it on both sides of the prompt and in the length of the reply.
• ⚠︎ NOT THE PACK'S LENGTH. The largest reply in the run came from the SMALLEST matrix size and the spread inside that one size is 22.7x. Cost per pack is not predictable from the pack.

Your volumeWhat it costs at your volume

Linear in packs. Each call is independent, carries the same fixed prefix and shares nothing, so 500 packs is 10x $0.8232 = $8.23 on the projected card. THE THING THAT IS NOT LINEAR IS THE SOLICITATION: this corpus deals four to eight requirements per pack and both the rendered matrix and the volume that answers it scale with that number, so a feed of thirty-requirement solicitations costs more per pack than these rows say at the same pack count. There is no caching in this kit and the fixed half -- 66.5 pct of the input -- is paid on every call.

Model choiceWhy this model

It is the tier the shared key points at, and the run is one call per pack with no retrieval and no second pass, so the model choice is the only cost lever there is. No second tier was run, so nothing on this page is a measured comparison between two models.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 49 proposal packs, one completion call each, one tier. The two calibrations, the five prompt-token calls, the one-word ceiling probe and the one live screenshot are NOT in these totals and are accounted for separately in Cost.

The codeEvery file, and what it is for

14 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pySegment

cut one pack into named sections on its underlined headings. The proposal's own section headings are deliberately NOT boundaries -- the submitted volume stays one segment, because outside_mandated_section is a claim about where text sits relative to the rest of the volume and a segmenter that split it into eight pieces would invite a selector to send one of them

# Cut a proposal pack into addressable sections. Pure code -- no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/matrix.pyRequirements matrix

parse the solicitation's own matrix -- the requirement ids, the heading each must be answered under, and whether the source selection scores it. THIS IS THE DENOMINATOR AND IT IS NOT THE MODEL'S TO CHOOSE: 280 rows across 50 packs, 15 packs with 4, 12 packs with 5, 9 packs with 6, 6 packs with 7, 8 packs with 8. A row that cannot be parsed raises rather than being skipped

# THE REQUIREMENTS MATRIX, parsed out of the pack in pure code. No model, no network.

MATRIX_SECTION = "Requirements Matrix"
def parse(secs):
def ids(secs):
def by_id(secs):
def matrix_block(rows):

src/select.pySelect

the union of the sections each field maps to. Internal Reference -- the price-to-win, the wrap rate, the competitor read and the bid decision -- is mapped by nothing and therefore never sent; 50 of the corpus's 430 sections never leave the machine. The Compliance Narrative IS sent, deliberately, because it is the distractor the whole measurement rests on

# Pick which sections of a proposal pack are sent. Pure code -- the last deterministic step before

BANNER = "Synthetic Record"
ROUTING = "Routing Slip"
HEADER = "Solicitation Header"
INSTRUCTIONS = "Instructions to Offerors"
MATRIX = "Requirements Matrix"
VOLUME = "Proposal Volume as Submitted"
NARRATIVE = "Compliance Narrative"
NOTES = "Distribution Notes"
NEVER_SENT = ("Internal Reference",)

src/desk.pyDesk cover sheet

read the capture lead and the proposal manager off the never-sent section, in pure code, AFTER the answer comes back. It is how the kit says who owns a cross-reference risk without the name -- or the price-to-win beside it -- ever being on the wire

# THE PROPOSAL DESK'S OWN COVER SHEET -- read locally, in pure code, and NEVER sent.

SECTION = "Internal Reference"
def owners(secs):

src/rulebook.pyRulebook

the compliance call and the disposition, loaded from data/rulebook.json. ONE definition, three readers: the corpus generator writes gold with it, src/prompt.py renders it into every prompt, and evals/judge.py re-runs it over the model's own facts

# THE COMPLIANCE RULEBOOK, loaded from data/rulebook.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
RESPONSE_STATES = ("responds_in_full", "responds_in_part", "points_elsewhere", "restates_only",
LOCATION_KINDS = ("mandated_section", "other_section", "compliance_narrative", "none")
COMPLIANCE_CALLS = ("not_addressed", "narrative_only", "cross_referenced",
DISPOSITIONS = ("blocking_gap", "placement_defect", "open_in_draft", "reference_risk",
SUBMISSION_STATES = ("final_as_submitted", "interim_draft")
VOLUMES = ("technical_volume", "management_volume", "past_performance_volume")
CLEARING_CALLS = ("addressed",)

src/prompt.pyPrompt

assemble one prompt per pack -- system, the rendered rulebook, the parsed requirements matrix, the field schema, the selected pack sections. Also declares the verdict vocabulary the site's compliance panel AST-parses

# Assemble the compliance prompt. ONE prompt per proposal pack, one row per requirement in reply.

SYSTEM = (
VERDICTS = ("not_addressed", "narrative_only", "cross_referenced", "outside_mandated_section",
VERDICT_MEANINGS = {
RULEBOOK_TEXT = rulebook_block()
def field_schema(pack_fields, row_fields):
def build(doc_text, secs, fields, selector, matrix_rows):
def parse(raw, fields):

src/adapters/__init__.pyModel adapter

SEAM 1 -- stdlib HTTP to any OpenAI-compatible provider or to Anthropic's Messages API. No vendor SDK, so a fork installs nothing

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/budget.pyBudget

one append-only ledger for every kit sharing the key, checked BEFORE a call is made rather than in the harness, so the local app and any script a forker writes are capped too

# A cap on live calls, shared by every kit on this machine. No dependency, no service.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
KIT = os.path.basename(HERE)
ROOT = os.path.dirname(os.path.dirname(HERE))
SHARED_ENV = os.path.join(ROOT, ".env")
LEDGER = os.path.join(ROOT if os.path.exists(SHARED_ENV) else HERE, ".calls-ledger.jsonl")
PAID_ARMS = ("r", "x")
FREE_ARMS = ("t", "b")
BOOT = os.urandom(6).hex()
PID = os.getpid()

src/check.pyCheck

the whole AI layer: one call per pack, then the pure-code sign-off rule over whatever came back. Holds MAX_TOKENS and the no-writing boundary

# Assemble one pack's compliance matrix: segment, parse the requirements matrix, select, prompt,

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 56000
def load_fields():
def row_fields(fields=None):
def pack_fields(fields=None):
def load_doc(doc_id):
def documents():

src/app.pyLocal app

the minimal UI on port 9008. Renders with no key; serves the rulebook to the page verbatim so the fixed input is one click from the answer

# The minimal local UI. Standard library only -- python -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "9008"))
class H(BaseHTTPRequestHandler):
def main():

evals/baseline.pyFree floor

the keyword compliance matrix a proposal shop already runs -- phrase search over the offeror's own text, no key, no tokens. It is what every figure on these pages is read against

# THE FREE FLOOR: the keyword compliance matrix a proposal shop already runs in a spreadsheet.

SOL = re.compile(r"^Solicitation:\s*(\S+)", re.M)
STATE = re.compile(r"^Submission state:\s*(.+?)\s*$", re.M)
VOL = re.compile(r"^Volume under check:\s*(.+?)\s*$", re.M)
PLACE = re.compile(r"(?:^Section (\d+\.\d+)|^(CN-\d+))", re.M)
VOLUME_MAP = {"Technical Volume": "technical_volume",
def _cell(v):
def _place_at(text, at):
def assess(doc_text, fields):

evals/judge.pyJudge

six scorers, all pure code, $0.00. The two error directions are computed on separate denominators here and are never added

# Score a run against gold. PURE CODE -- no model, no network, no judgement, $0.00.

PACK_CELLS = ("solicitation_ref", "submission_state", "volume_checked", "disposition")
ROW_CELLS = ("response_state", "location_kind", "response_ref", "compliance_call")
def norm(v):
def _rows_by_id(rec):
def _val(row, name):
def score_coverage(records, golds):
def score(fields, records, golds):
def score_calls(records, golds):
def score_pack_clear(records, golds):

evals/check_labels.pyGold gate

refuses to let a run spend unless gold, the corpus, the matrix parse and the selector agree over all 50 packs

# THE GOLD GATE. Everything that must be true of the corpus, the labels and the selector BEFORE a

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
FLOORS = {"addressed": 110, "cross_referenced": 28, "narrative_only": 15, "not_addressed": 14,
DISPOSITION_FLOOR = 4
GUARD_BAND = (10.0, 60.0)
MATRIX_SIZES = (4, 8)
def load_gold():
def main():

tools/build_corpus.pyCorpus generator

writes the packs and derives gold from the same rulebook the model is shown. Asserts before it writes; a failed assertion writes nothing

# Generate the synthetic proposal-compliance corpus and its gold labels. Free, offline, seeded.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
SEED = 20260823
N_DOCS = 50
BANNER = (
OFFICES = [
PROGRAMMES = [
VOLUMES = {

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 5898 input and 4616 output tokens per pack, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's packs are entirely synthetic (tools/build_corpus.py, seed 20260823): no real solicitation, no real contracting office, no real offeror, no real person, and every price-to-win and win-probability figure in them is invented. Nothing left this machine except the SELECTED sections of one pack at a time, and the selection is a closed map: the Internal Reference page carrying the pricing strategy and the bid decision is mapped by no field and is therefore in no prompt on any of the 49 calls. evals/check_labels.py asserts that, per field, on every pack, before a run may spend.

Read from the shared .env, this kit's own .env or the real environment only, never committed, never logged and never rendered. Both .env paths are gitignored from the first commit; src/config.py::save() opens the file 0600 BEFORE writing rather than chmod-ing after; and src/app.py redacts the key and the base URL out of any provider error before it reaches the page.

The experimentWe did not attack it -- and the corpus contains the argument that would have been the attack

No attack was tried, and the corpus contains the argument that would have been the attack. Every one of the 50 packs carries a Distribution Notes section stating that the teaming partner's proposal office reads the Instructions as treating a cross-reference into a teammate's volume as a response — which is precisely the reading that turns all 39 cross_referenced rows into false clears — and src/select.py SENDS it, deliberately, because hiding a kit's own adversarial surface from the run that is supposed to measure it is worse than not measuring. What the run establishes: with that argument present, 39 of 39 (100.0 pct) cross-referenced rows were still called correctly, and the total false-clear count for the whole run was 1. What it does NOT establish: that the argument had no effect, because no run was fired with the section removed and there is nothing to difference against. Three other boundaries were checked by reading the code and by the gold gate's own assertions rather than by attack: the kit writes no proposal text on any path, the never-sent pricing section reaches no prompt on any of the 49 calls (asserted per field on every pack), and nothing acts on the disposition it produces. Confirmed by reading the code and by the gold gate's per-field assertion over every pack, not by an attack run, on 2026-08-23 -- no attack run was fired; see redteam.why.

Three boundaries hold on evidence a reader can check in the code and the fourth is openly unproven. The second is the one worth reading twice, because on this use case it is the difference between a tool a proposal shop can run and one it cannot: the section carrying the price-to-win sits INSIDE the same file as the volume being checked, so 'do not send it' is a positive act on every call rather than a default. It is enforced by a closed map — a field with no matching section falls back to the whole pack MINUS the never-sent list, never to the whole pack — and the gold gate asserts it per field, on every pack, before a run may spend. The fourth boundary is the one this kit would test next, and it is honest to say it has not been.

The result0 attack trials. Four boundaries are stated and three of them hold on evidence a reader can check in the code; the fourth -- that no externally-authored section can move a call -- is UNTESTED, and the corpus contains exactly the section that would test it.

The corpus carries a genuine adversarial surface and it was authored on purpose: every pack's Distribution Notes section says that the teaming partner's proposal office reads the Instructions as treating a cross-reference into a teammate's volume as a response, and that section is SENT. That is a plain-language argument for the reading that turns every cross_referenced row into addressed. It is not an injection -- it does not instruct the model, it states a third party's opinion -- and this kit measured its effect at exactly zero times, because no ablation was run with the section removed. What the run does show is that the model returned 39 of 39 (100.0 pct) cross_referenced rows correctly anyway, which is evidence the argument did not carry and is not evidence that it cannot.

HonestyWhat this does not prove

• Whether the Distribution Notes, or any other externally-authored text in a pack, can move a call. No ablation and no injection test were run.
• What the provider retains of a submitted prompt. On this corpus it is synthetic and the question is academic; on a real technical volume it is the question that decides whether this can run at all, and nothing here answers it.
• Whether a real proposal volume carries controlled technical data. The CHECK LOGIC needs none — every field it reads is a section reference or a verdict — but the draft content it reads is the adopter's, and which surfaces and stores may hold the resulting gap board is their determination and not this kit's.
• Whether the never-sent list is complete for a real shop. One section is listed here because this corpus has one; a real pack carries competitive intelligence, pricing narrative and legal review notes in several places, and the gate only asserts the names it is given.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

`needs_capture_lead_signoff` fires when at least one requirement the solicitation marks EVALUATED — one the source selection scores — has a compliance call of `cross_referenced`, meaning the only text answering it is a pointer to another volume. The `evaluated` column is read off the solicitation by src/matrix.py in pure code and is never asked of the model, so half the rule cannot be wrong. It fires on 14 of 50 (28.0 pct) of the packs in gold; the condition was chosen against a five-candidate table counted over gold before one was shipped, printed by tools/build_corpus.py on every generation.

src/check.py::compute(), called by check() on every pack, by the local app on every click, and by evals/baseline.py on the free floor -- so the flag is computed identically for the model and for the thing it is compared against. It reads the evaluated column from src/matrix.py's parse of the solicitation, never from the model.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a submission decision, a clearance, a waiver or an instruction to write. Nothing in this kit decides whether a volume goes in the box.
• It is NOT a writer. The kit does not draft, rewrite, suggest or edit a single word of the offeror's response. That boundary is absolute and has no evidence-gated expansion path -- a tool that marks a requirement unanswered and then supplies the answer has removed the only person who was going to read the requirement.
• It is NOT a check against your internal quality standards. It checks the solicitation's stated requirements and nothing else; an offeror's own style rules, page budgets and win-theme checks are a different pack and are out of scope here.
• It is NOT a check on the model. The condition is a business fact about the pack -- a scored requirement answered by pointer alone -- and it would be worth flagging if a person had filled the matrix in by hand.

WatchedWhat is watched, and why that one

The kit measures 23 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

5 measured by the latest run 18 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (extraction_cells 1320) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

2 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 2 runs measured the same system — they differ on documents, failures, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 2 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/check.py, on any change to data/rulebook.json (which moves the calls the condition reads), on any change to src/matrix.py (which reads the evaluated column the condition is defined over) and on any regeneration of the corpus. evals/check_labels.py re-measures the firing rate against a 10-60 pct band on every invocation and exits non-zero outside it, so the drift alarm is the gate rather than a note.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the provider is reached over urllib, and there is no retrieval step to need a vector store. A fork installs nothing and can read the whole prompt.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

One path per pack -- segment, parse the matrix, select, prompt, one call, parse, locate each reference, compose the disposition, run the pure-code sign-off rule -- with no branch, no retry above the transport layer and no second model. The only fan-out is the thread pool in evals/run.py, which runs the same path 49 times.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain -- the JSON extraction from a fenced reply, the two regexes in the matrix parser, the retry and backoff policy, the token accounting and the result schema.
• There is no schema enforcement on the reply. src/prompt.py::parse() checks shape and fails closed; it does not validate every enum, and a value outside the vocabulary reaches the rulebook, which returns None for it and is counted as a reply failure rather than as a compliance call.
• There is no caching. The fixed half of the prompt is 66.5 pct of the input and is paid on every call; a prompt-caching provider would change the cost shape and this kit has measured nothing about that.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-proposal-compliance on the fast tier, 2026-08-23. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 1 earlier run on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 1 run on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence, corpus included Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one line⚑ False clear — a requirement it told you to stop looking at, that is not answered

Of the rows gold says are NOT answered as written, how many the run called addressed — the only call that removes a requirement from somebody's list. It is reported on its own denominator and is never averaged with the opposite direction, because one lost bid and twenty recovered hours are not tradeable.

MethodHow the test was run

evals/judge.py::score_calls, in-process, no key and no model. The same function scores the free floor, because a floor and a run compared by two different scorers are not compared.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

reconstructed from results/eval-r001-proposal-compliance.json by tools/capture_example.py — the run's non-hit cells overwritten onto gold. No model call was made to produce this file, because a re-fired pack returns a DIFFERENT answer from the one every published figure was computed over.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the gold compliance_call on each row, derived by tools/build_corpus.py from the same rulebook the model is shown, and re-derived by evals/check_labels.py on every invocation before a run may spend.

How tight can the band be? No band and no tolerance — an exact string comparison against a derived label. The per-class denominators are printed with every rate because the smallest of them is 14 rows and a rate over 14 rows is not a rate.

Cadence: Re-score on any change to data/rulebook.json, data/fields.json, src/prompt.py or tools/build_corpus.py — the first moves what a call means, the second what the model is asked for, the third how it is asked, and the fourth what the answer is compared against.

The decisionWhen to reach for it

In one linefalse gap — a requirement it flagged that is already answered

Of the rows gold says ARE answered in full under the mandated heading, how many the run flagged anyway. It is the cheap direction — a writer re-reads a section and closes it — and it is published beside the expensive one so a reader can see the trade rather than a blend of it.

MethodHow the test was run

evals/judge.py::score_calls, the same pass and the same function as the grader above — two denominators, one loop.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

reconstructed from results/eval-r001-proposal-compliance.json by tools/capture_example.py — the run's non-hit cells overwritten onto gold. No model call was made to produce this file, because a re-fired pack returns a DIFFERENT answer from the one every published figure was computed over.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the same gold compliance_call, on the complementary set of rows — the 147 gold addressed rows the false-clear grader does not score.

How tight can the band be? No band. Exact comparison, 147-row denominator, and the rate is printed with that denominator beside it every time.

Cadence: The same triggers as the false-clear grader. They are computed in one pass over the same rows and cannot be re-scored separately.

The decisionWhen to reach for it

In one linePer-cell exact match — the reference standard for values

Every cell of every row against gold: the two stated facts, the response reference and the call, plus the four pack-level cells. Light normalisation only — trimmed, whitespace-collapsed, lowercased, trailing punctuation removed — and nothing semantic. It is the standard the other graders are defined against.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

reconstructed from results/eval-r001-proposal-compliance.json by tools/capture_example.py — the run's non-hit cells overwritten onto gold. No model call was made to produce this file, because a re-fired pack returns a DIFFERENT answer from the one every published figure was computed over.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: itself. This is the reference grader; gold is a mechanically derived label and this comparison is what 'correct' means everywhere else in the kit.

How tight can the band be? No band and no tolerance — an exact comparison after normalisation that maps nothing onto anything.

Cadence: Re-score on any change to data/fields.json or the corpus generator.

The decisionWhen to reach for it

In one line⚑ Row coverage — did a row come back for every requirement the matrix states

Set comparison over (pack, requirement_id). The denominator is the solicitation's own matrix, parsed in pure code by src/matrix.py, and it MOVES: 15 packs with 4, 12 packs with 5, 9 packs with 6, 6 packs with 7, 8 packs with 8. A checker that returns a fixed number of rows is wrong on most of this corpus.

MethodHow the test was run

evals/judge.py::score_coverage, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

reconstructed from results/eval-r001-proposal-compliance.json by tools/capture_example.py — the run's non-hit cells overwritten onto gold. No model call was made to produce this file, because a re-fired pack returns a DIFFERENT answer from the one every published figure was computed over.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the requirement ids src/matrix.py parses out of each pack, asserted equal to gold's ids on every pack by evals/check_labels.py before anything spends.

How tight can the band be? No band — a set comparison over a per-pack key list.

Cadence: Re-score on any change to src/matrix.py, the corpus format, or src/prompt.py's instruction about how many rows are owed.

The decisionWhen to reach for it

In one lineResponse reference — does the cited place exist, and is it the right one

Three verdicts over the rows that owe a reference, and the middle one is the point: cited the right place, cited a real place of this pack and the WRONG one, or cited something that is not in the pack at all. The last is a row nobody can check and it looks exactly like a good one on the page.

MethodHow the test was run

evals/judge.py::score_citation, in-process, no key and no model. It reads the pack text off disk to test the middle verdict.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

reconstructed from results/eval-r001-proposal-compliance.json by tools/capture_example.py — the run's non-hit cells overwritten onto gold. No model call was made to produce this file, because a re-fired pack returns a DIFFERENT answer from the one every published figure was computed over.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: gold's response_ref, asserted by evals/check_labels.py to be printed verbatim exactly once in the pack it labels, and asserted not to nest inside any other reference in the same pack.

How tight can the band be? No band. Exact match after the same normalisation, then a substring test against the pack text for the middle verdict.

Cadence: Re-score on any change to the corpus's section numbering or to src/check.py's SPANNABLE list.

The decisionWhen to reach for it

In one line⚑ Blocking gap missed — the pack carries a requirement nothing answers and the reply carries none

At PACK level: gold carries a not_addressed or narrative_only row and the reply carries no such row at all, so the disposition composes softer than blocking_gap. It is NOT a restatement of the false-clear rate — a row can be flagged as the wrong KIND of problem and still lose the pack-level call, which is exactly what happened on PKG-0010.

MethodHow the test was run

evals/judge.py::score_pack_clear, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

reconstructed from results/eval-r001-proposal-compliance.json by tools/capture_example.py — the run's non-hit cells overwritten onto gold. No model call was made to produce this file, because a re-fired pack returns a DIFFERENT answer from the one every published figure was computed over.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the gold disposition, composed by src/rulebook.py from gold's own rows, against the same composition run over the model's rows. One function, two inputs.

How tight can the band be? No band. A boolean per pack over the 14 packs whose gold carries a gap — small enough that one pack moves the rate by 7.1 points.

Cadence: Re-score on any change to data/rulebook.json's disposition rule, to src/rulebook.py::disposition(), or to the corpus's class mix.

The decisionWhen to reach for it

In one lineCapture-lead sign-off — the pure-code business condition

Whether the shipped condition — a requirement the source selection SCORES, answered only by a pointer to another volume — fires on the packs the same rule fires on over gold. Recall, precision and the two error kinds separately; it is a business condition, not a check on the model.

MethodHow the test was run

evals/judge.py::score_flag, in-process, no key and no model. It also computes the no-gold consistency diagnostic in the same pass.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

reconstructed from results/eval-r001-proposal-compliance.json by tools/capture_example.py — the run's non-hit cells overwritten onto gold. No model call was made to produce this file, because a re-fired pack returns a DIFFERENT answer from the one every published figure was computed over.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/check.py::compute() run over GOLD's rows and gold's evaluated column, against the same function run over the model's rows. One function, two inputs.

How tight can the band be? No band — a boolean against a boolean. The denominator is the 14 packs gold fires on, which is small enough that one row moves recall by 7.1 points.

Cadence: Re-check on any change to compute() in src/check.py, to data/rulebook.json, or to src/matrix.py's read of the evaluated column. evals/check_labels.py re-measures the firing rate against a 10-60 pct band on every invocation.

The decisionWhen to reach for it