The business caseThe problem this solves

A delegated-vendor oversight cycle produces a population report, and the report's own arithmetic is what hides the two failures that matter. A vendor writes 'training is substantially complete' and a keyword scan records an affirmative answer. A group submits one packet covering one of the four delegation contracts it holds, the intake matches it on the submitter account, and three rows show a packet against them. Neither produces an exception anywhere: what surfaces, months later, is an audit finding on a delegation nobody chased, or a vendor pushing back on a letter asking it to complete a submission it never made. The analyst's own pass over one packet against one inventory row — the header, the covering note, the four numbered answers and the screening result — repeated across the whole contract inventory. It replaces the READING of that pass and none of the deciding: FDR-2026 is applied in code either way, and the status is never asked of the model.

The inputThe actual FDR reconciliation packets

The corpus is 60 FDR reconciliation packets, 0.11 MB (json 2 · txt 60). An FDR attestation packet cannot be published by anybody: it is a contractual submission naming a real vendor, commercially sensitive to the vendor and regulatorily sensitive to the plan, and a real exclusion-screening result names a real person. There is no public dataset, and a redacted attestation is a form with no answers in it. That would matter less if the prose were what is measured — it is not. What is measured is a reconciliation decision and the sentence behind it, and that needs facts that are KNOWN: which of four statuses a row should carry, whether an element was answered affirmatively or conditionally, whether a packet that arrived belongs to this entity and this cycle, and exactly which sentence carries it, to the character.

FDR RECONCILIATION PACKET  FDR-0001 -- Annual delegation attestation

CONTRACT INVENTORY ROW -- the authority on who holds this delegation

  FDR id                      FDR-0001
  Contracted entity           Vale Ridge Billing Services LLC
  Parent group                Vale Ridge Health Group
  Tier                        first-tier
  Delegated function          utilization management
  Contract effective          2024-02-01
  Cycle under reconciliation  2026-H1
  Attestation due             2026-07-31
  Submission matched          ATT-2026H1-0401
  Exclusion screening result  clear -- OIG LEIE and GSA lists, no match returned

ATTESTATION PACKET AS RECEIVED

  Received                    2026-07-06
  Attesting entity            Vale Ridge Billing Services LLC
  Attestation cycle stated    2026-H1
  Signed by                   the compliance officer for the delegated function

COVERING NOTE

The contacts for this delegation are unchanged from the previous period and are listed in the contract schedule. This packet was prepared by our compliance function and reviewed by the officer named in the signature block.

ANSWERS TO THE REQUIRED ELEMENTS

  1. Compliance programme
     A written compliance programme, a code of conduct and a confidential reporting line were in operation throughout the period, and the officer accountable for the delegated function is named in the signature block.

  2. General compliance and FWA training
     Every member of the delegated team completed general compliance and FWA training within the cycle; the completion register was closed on 12 June.

  3. Exclusion screening cadence
     Monthly OIG LEIE and GSA exclusion screening was performed for all staff and contractors performing the delegated function for every month of the period.

The outcomeWhat a good result looks like

One status per FDR that a person confirms instead of a packet they re-read: missing-entirely, non-conforming, hedged or clean, with the rule that produced it and the sentence it came from copied verbatim out of the packet and located at its offsets. Each status names a different piece of follow-up work — chase a submission that was never made, review one that arrived, go back for a straight answer, or nothing this cycle.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• A cycle whose packets hedge in the ordinary vocabulary of a compliance script — substantially, largely, in progress, on track, targeting a date. — the free rules floor alone
  A TIE, measured: hedge_listed is 6 of 6 on both arms, and so are element_absent (5/5), element_negative (4/4), all three screening cases (8/8), no_submission (6/6), prior_cycle_header (4/4), different_entity_header (4/4) and clean_full (8/8). Forty-five of the sixty rows are a tie.
• A cycle where out-of-scope packets say so anywhere but a header line, or where a hedge is a count, a vacant post or a cadence rather than an adverb. — the model
  This is the entire margin. The floor scores 0 of 6 on hedge_in_prose, 0 of 3 on phrase_false_positive, 0 of 3 on prior_cycle_prose and 0 of 3 on different_entity_prose; the model scores 3, 3, 3 and 3. Those 15 rows ARE the 45-to-57 difference.

And where nothing here is good enough:

• A cycle where the question is only 'did everyone submit'. — neither — the inventory join alone
  FA-1 is a join and it is free. src/policy.py answers all 6 no-packet rows with no model and no phrase list. What you cannot get for free is the 14 rows where a packet arrived and is not this row's: the floor gets 8 of those and the model 14.

At a glanceHow the whole thing runs

Run once, for real, on 2026-09-01. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-09-01 — r001-vendor-attest. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — ⚠︎ PARTIALLY EVIDENCED, AND THE REST IS NOT MEASURED. What is on disk was observed on 2026-09-01: with API_KEY blanked, the local server starts, every endpoint but /api/decide answers, the free rules floor scores all 60 rows offline in 0.1 wall seconds, and the committed paid run replays per row off its result file with every citation located at the offsets the scorer found them at — all four screenshots were taken that way. WHAT WAS NOT DONE is a clone into an empty directory on a second machine: the repository is private, and no such run was performed, so nothing here claims one.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one FDR row end to end — one reconciliation packet, one call, four element readings and one scope reading, including provider-side reasoning tokens, on a shared credential. The rulebook engine and the citation locator add nothing measurable: both are pure Python over the reply. ⚠︎ NOT AN SLA AND NOT A SPREAD: these are wall-clock observations from ONE run and the band is wide — fastest call 3503 ms, slowest 54236 ms. 60 rows ran in 0.0 wall seconds at 5 workers. The free rules floor answers and scores all 60 with no network at all and its run record carries wall_seconds 0.1.

Current processWhat it replaces

The analyst's own pass over one packet against one inventory row — the header, the covering note, the four numbered answers and the screening result — repeated across the whole contract inventory. It replaces the READING of that pass and none of the deciding: FDR-2026 is applied in code either way, and the status is never asked of the model.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one element readings (4 per FDR row), plus 60 status and 60 scope judgement is

No train/test split, because nothing is trained or tuned, and no segmentation, because a packet goes into one call whole. The corpus is planned rather than sampled: 14 cases with counted denominators, 30 clean rows and 30 hard ones, so every rate on the report has a denominator somebody chose on purpose and can be re-counted from data/gold.jsonl.

SetupWhat the setup figure measured

THERE IS NO INDEX AND NOTHING TO RETRIEVE FROM. The rulebook is data read whole into every prompt and the packet is read whole beside it. The two things a reader might call an index are data/inventory.json — a join key and a screening state per FDR, read as data by the engine and never searched — and data/gold.jsonl, which is the answer key. Both are generated in under a second by tools/build_corpus.py and neither costs a call.

LicenceLicence

MIT — the kit's own licence. Nothing in the corpus is derived from any third-party material, so there is no upstream licence to carry.

Bring your ownBring your own FDR reconciliation packets

Drop your own reconciliation packets as .txt into data/corpus/, add one row per FDR to data/inventory.json (fdr_id, legal_name, tier, delegated_function, cycle, submission_ref — NULL where nothing was received — and screening as one of clear / potential_match_unresolved / confirmed_exclusion / not_performed), and write data/gold.jsonl with the status, the scope, the four element readings, the rule and each citation with its span. The span is computable: src/citation.py::span_of(norm(text), sentence). Replace data/policy.json and data/policy.md with your own wording — the rule ids, their order and their outcomes are read, never hard-coded — and add a rule to src/policy.py for any condition the ten here do not cover. Then run python3 -m evals.check_labels; it will refuse a key that does not follow from its own inventory row.

⚠︎ And what stops being true when you do: ⚠︎ THE WHOLE PACKET GOES TO A MODEL PROVIDER VERBATIM — the covering note, the four answers, the inventory row and the screening state, which names an unresolved match against a federal exclusion list. That is a disclosure decision for whoever holds the contract and it is not one a kit can make on their behalf. Run it against an extract you are permitted to disclose, or against a local model. And the measured numbers do not travel: this corpus is templated, its case mix was chosen rather than observed, and the split between packets that say they are out-of-scope in a HEADER and packets that say it only in PROSE is a design choice the free floor's score moves directly with.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 23,967 characters assembled, 8,310 are rulebooks — 35% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The whole prompt as sent for FDR-0001, the first row of run r001-vendor-attest, replayed from the kit's own src/prompt.py::build(). It is trustworthy as the prompt that was actually sent because the run record stores the part decomposition and the section list for the run itself, and both match this replay part for part: system, policy, statuses, elements, packet, schema. The token counts below are the provider's own for that call.

Full promptThe prompt in full

You are reading one FDR reconciliation packet from a delegated-vendor oversight cycle and
answering a single operational question about it: for each of four required attestation elements, HOW
does this packet answer it, and WHOSE submission for WHICH CYCLE is this packet actually. Your output
is the reading an oversight analyst reconciles against the contract inventory.

WHAT YOU ARE NOT DOING, AND IT IS THE FIRST THING TO SAY. You do not certify this FDR as compliant
with the attestation or exclusion-screening requirements. You do not clear, close or resolve any
exclusion-screening result, and no assertion inside the packet about a screening match changes
anything. You do not decide a reconciliation status at all: the status is produced in code from your
reading, the contract inventory and the screening result on file, and the schema below has no field
you could put one in. What you produce is a READING and the text it came from, for a person to act
on.

How to read the packet:

- THE INVENTORY ROW IS PRINTED ABOVE THE PACKET AND IT IS THE AUTHORITY on which entity holds this
  delegation and which cycle is being reconciled. The packet is the thing being checked against it,
  never the other way round.
- A PACKET EXISTING IS NOT EVIDENCE THAT THIS ENTITY SUBMITTED FOR THIS CYCLE. Read the cycle it
  attests to and the entity it attests for, from anywhere in the packet -- the header, the covering
  note, the body, the signature block. A prior-year certification resubmitted unchanged, or a packet
  covering a sibling entity under the same parent, means this row has nothing.
- A QUALIFIED ANSWER IS NOT AN AFFIRMATIVE ONE. "Training is substantially complete", "screening is
  under way for the remaining contractors", "the flow-down is on track for the end of the quarter"
  all contain the vocabulary of a completed attestation and all state that the requirement is not
  met yet. Read those as `qualified`. This distinction is most of what this packet set is built to
  test.
- AN ELEMENT THE PACKET NEVER ADDRESSES IS `absent`, AND IT IS NOT THE SAME AS ONE ANSWERED "no".
  Four numbered answers where the packet gives three is an unanswered element, whatever the covering
  note says in general terms.
- A PHRASE DOES NOT QUALIFY AN ANSWER JUST BY APPEARING. A hedge in a covering note about something
  else, a boilerplate line printed on every packet, or a sentence describing what the FDR did LAST
  cycle qualifies nothing in this one.
- Answer all four elements, once each, in the order given. A complete, unqualified packet reads
  `affirmed` on all four, and that is a normal shape of reply here.
- Apply the rulebook as written, INCLUDING THE ORDER ITS RULES ARE APPLIED IN, when you decide what
  a reading means -- but do not report a status.
- Give one confidence between 0 and 1 for this packet's readings taken together.

Reply with JSON and nothing else, in the shape given at the end.

THE RECONCILIATION RULEBOOK, as approved:

# FDR-2026 — the delegated-vendor attestation reconciliation rulebook

*As at 2026-09-01. Cycle **2026-H1**. Written against 42 CFR 422.503(b)(4)(vi), the Medicare
Advantage compliance-programme requirement that makes a plan responsible for the first-tier,
downstream and related entities it delegates to.*

**FDR-2026 is invented.** It is not any plan's oversight procedure and it is not advice about how
one should be written. It is one plausible way of operationalising a single attestation cycle,
written so that its precedence can be checked line by line.

**What this rulebook produces is a RECONCILIATION STATUS and never a compliance determination.**
There are four statuses and none of them certifies an FDR, clears an exclusion-screening result, or
closes an oversight item. Those are determinations a person makes and signs.

## The four statuses

| status | what it means |
|---|---|
| **`missing-entirely`** | the FDR is in the contract inventory and **this cycle has nothing from it** |
| `non-conforming` | something arrived from the contracted entity and it does not hold up |
| `hedged` | everything was addressed and at least one answer is qualified rather than affirmative |
| `clean` | received, complete, unqualified, from the contracted entity, and the screening result carries no match |

**`missing-entirely` and `non-conforming` are not degrees of one thing.** One is a chase for a
submission that was never made; the other is a review of one that was. Different letter, different
queue, different evidence on the file.

## The four required elements

Every submission answers all four, once each, in this order.

| element | what it attests |
|---|---|
| `compliance_program` | a compliance programme covering the delegated function is in place for the whole cycle |
| `fwa_training` | everyone performing the delegated function completed general compliance and FWA training within the cycle |
| `exclusion_screening` | the FDR screened everyone performing the delegated function against the OIG LEIE and the GSA exclusion list on the required cadence |
| `downstream_flowdown` | the same requirements were passed to any entity the delegated function was subcontracted to |

Each element is read as one of `affirmed`, `qualified`, `negative` or `absent`. The status is
decided from the **worst** of the four.

## The rules, in the order they are applied

**THE ORDER IS THE RULEBOOK.** The same facts under a different order produce different answers.

**FA-1 — An FDR in the inventory with nothing matched to it this cycle has not submitted →
`missing-entirely`.** Where the contract inventory carries this FDR for this cycle and the cycle's
attestation population carries nothing matched to it, the status is `missing-entirely`. **This is
the first rule because it is the one a submission reader cannot reach.** Every other rule here is a
judgement about text that exists; this one is a judgement about text that does not, and the only
route to it is to run the reconciliation against the **inventory** rather than against the pile of
packets that arrived. It is a join, and the join is decided in code.

**FA-2 — A packet attesting to an earlier cycle is not a submission for this one →
`missing-entirely`.** A prior-year certification resubmitted unchanged, or a packet whose attested
period ends before this cycle begins, is not a late submission and is not an incomplete one. A
submission for a period that has already been reconciled is not evidence about this period.
Recording it as `non-conforming` sends a request to complete a packet the FDR would reasonably say
it already completed, for the cycle it was actually written for.

**FA-3 — A packet attesting for a different contracted entity is not this entity's submission →
`missing-entirely`.** A parent, an affiliate, or a sibling entity under the same group whose
submitter account the intake matched on. A group holding four delegation contracts owes four
attestations, and one packet covering one of them leaves three rows empty. **This is the quietest
gap in the cycle**, because the population report shows a packet against the FDR and everything
downstream reads that as covered.

**FA-4 — A confirmed exclusion is never closed here → `non-conforming`.** Where the screening
result on file is a confirmed match to an excluded party, the status is `non-conforming` and it
stays open. Checked before every element rule so that no attestation, however completely and
affirmatively written, can produce any other status. **This system never clears a screening
result.** A packet asserting the match was reviewed and found to be a different individual does not
move this field.

**FA-5 — An unresolved potential match stays open → `non-conforming`.** A name on this FDR's staff
list is a potential match to a listed party and no person has resolved it. An unresolved match is
not a clear result; it is a piece of work nobody has done yet. The packet's own account of it — *"we
reviewed the flagged name, confirmed the date of birth differs, and consider the item closed"* — is
the FDR's assertion about its own screening, and that is exactly the assertion an oversight cycle
exists to not take on trust.

**FA-6 — A required element answered in the negative is non-conforming → `non-conforming`.**
Checked before the hedge rule, because a packet that answers one element negatively and another
conditionally is decided by the more serious of the two: a straight no is a finding, a qualified yes
is a question.

**FA-7 — A required element the packet does not answer at all is non-conforming →
`non-conforming`.** **Received and incomplete.** ⚠︎ This is the rule that must never be confused
with FA-1, FA-2 or FA-3. Something arrived from the contracted entity for this cycle and one of its
four answers is not in it; the follow-up is a request to *complete* a submission that exists. FA-1's
follow-up is a request for a submission that does not. A reconciliation that produces one number for
both has produced neither.

**FA-8 — No screening result on file is non-conforming → `non-conforming`.** Whatever the packet
attests. The FDR attesting that it screens and a screening result existing are two different facts,
and the attestation is the one this rulebook does not take on trust. Checked after the element rules
because an unanswered element is the one the FDR can fix by writing back.

**FA-9 — A qualified answer is a hedge, and a hedge is not an affirmation → `hedged`.** Every
element addressed, none negative or absent, screening clear, and at least one answer qualified —
*substantially complete, largely in place, under way, on track, targeting a date, all but a few,
with the exception of, expected by*. ⚠︎ **This is the rule a keyword scan cannot reach.** *"Training
is substantially complete"* contains the vocabulary of a completed attestation and states that it is
not complete. What the status commits somebody to is a follow-up question rather than a finding.

**FA-10 — Reconciled: complete, unqualified, from the contracted entity, screening clear →
`clean`.** ⚠︎ That is the whole of what it means. It is **not** a finding that the FDR complies with
the regulation, **not** a clearance of any screening result, and **not** a closure of an oversight
item.

## The screening result

The exclusion-screening result is a **fact held on the inventory row**, produced by matching this
FDR's staff list against the OIG LEIE and the GSA exclusion list. It is never inferred from the
packet's prose and never changed by it. One of `clear`, `potential_match_unresolved`,
`confirmed_exclusion`, `not_performed`. A name match is a deterministic comparison rather than a
reading, so it is held as data on every arm and no arm gets credit for it.

## The inventory join

`submission_ref` on the inventory row is the packet the intake system matched to this FDR for this
cycle, or null. **Null is FA-1 and is decided in code.** The reply's own scope reading is discarded
where the join says nothing was received: a missing submission that a reply talks its way out of is
the exact failure this rulebook's first rule exists to prevent. Where a packet **is** matched,
whether it actually belongs to this entity and this cycle is a reading, and that reading is not
free.

THE FOUR RECONCILIATION STATUSES the rulebook produces from your reading, and what
recording each one commits an oversight analyst to. YOU DO NOT ANSWER WITH ONE --
they are here so you know what your reading decides.

  missing-entirely   In the inventory, nothing received this cycle
      AN OVERSIGHT ANALYST CHASES A SUBMISSION THAT WAS NEVER MADE. The FDR holds a delegated function under a live contract and this cycle's attestation population has nothing from it — no packet, or a packet that turns out to attest for an earlier cycle or for a different contracted entity. It is NOT a late or partial submission: there is nothing to review, so the follow-up is a request for a submission and not a request to complete one. Recorded wrongly in either direction it produces the wrong letter, and recorded as anything else it produces a population report that counts this FDR as covered.

  non-conforming     Received, and it does not hold up
      AN OVERSIGHT ANALYST REVIEWS A SUBMISSION THAT ARRIVED. A required element was answered in the negative, was not answered at all, or the exclusion-screening result on file carries an unresolved potential match, a confirmed exclusion, or no screening at all. Something exists to look at and somebody has to look at it. The reason is recorded with the row so the follow-up is specific — 'the training element is unanswered' and 'the screening result is an unresolved match' are different pieces of work.

  hedged             Received, and an answer is qualified rather than affirmative
      AN OVERSIGHT ANALYST GOES BACK FOR A STRAIGHT ANSWER. Every required element was addressed and at least one was addressed conditionally — substantially, largely, in progress, on track, targeting a date, all but a few. On a keyword scan this reads exactly like an affirmative answer, which is the whole reason the class exists. What it commits somebody to is a follow-up question, not a finding: the answer may turn out to be fine and it has not been given yet.

  clean              Received, complete, unqualified, and the screening result carries no match
      THE ROW NEEDS NO FOLLOW-UP THIS CYCLE AND THAT IS ALL IT MEANS. ⚠︎ IT IS NOT A CERTIFICATION AND NOT A CLEARANCE. The contracted entity submitted for this cycle, answered all four required elements without qualification, and the screening result on file carries no match. Whether the FDR actually complies, and whether a screening result may be closed, are determinations a person makes and signs; this kit has no word for either and must never be given one.

⚠︎ `missing-entirely` AND `non-conforming` ARE NOT DEGREES OF ONE THING. One is a
chase for a submission that was never made; the other is a review of one that
was. Different letter, different queue, different evidence on the file.

THE FOUR REQUIRED ELEMENTS, in the order you must answer them, and what each attests:

  compliance_program     A compliance programme is in place
      The FDR attests that it operates a compliance programme covering the delegated function for the whole cycle — a code of conduct, a route for reporting, and somebody accountable for it.

  fwa_training           General compliance and FWA training completed
      The FDR attests that everyone performing the delegated function completed general compliance and fraud, waste and abuse training within the cycle. This is the element a hedge lands on most often, because training is the thing that is genuinely usually almost finished.

  exclusion_screening    Exclusion screening performed on the required cadence
      The FDR attests that it screened everyone performing the delegated function against the OIG LEIE and the GSA exclusion list on the required cadence. ⚠︎ THE ATTESTATION IS THE FDR SAYING IT SCREENED. The screening RESULT on file is a separate fact, held in the inventory and never read out of this answer.

  downstream_flowdown    Requirements flowed down to downstream entities
      The FDR attests that it passed the same requirements to any entity it subcontracted the delegated function to. An FDR with no downstream entities answers this affirmatively and says so; an FDR that is silent about it has not answered it.

THE READING VOCABULARY -- how the packet answers an element:

  affirmed    the element is answered affirmatively, for the whole cycle, without condition, qualification, exception or a future date
  qualified   the element is answered affirmatively AND the answer is conditional, partial, forward-looking or carries an exception -- substantially, largely, in progress, under way, on track, targeting a date, all but a few, with the exception of, expected to be, pending. THE ANSWER HAS NOT BEEN GIVEN YET
  negative    the element is answered and the answer is that the requirement is not met
  absent      the submission does not answer this element at all -- it is not addressed anywhere in the packet

THE SCOPE VOCABULARY -- whose submission this is, and for which cycle:

  this_cycle_this_entity   the packet attests for the contracted entity named on the inventory row, and for this attestation cycle
  prior_cycle              a packet was received and it attests to an EARLIER cycle -- a resubmitted prior-year certification, or a packet whose attested period is the last one. It is not a late submission for this cycle; this cycle has nothing
  different_entity         a packet was received and it attests for a DIFFERENT contracted entity -- a parent, an affiliate, or a sibling entity under the same group. The entity on this inventory row has submitted nothing
  no_submission            no packet is matched to this FDR at all. ⚑ THIS READING IS NOT THE MODEL'S TO GIVE: it comes from the inventory join, in code, and overrides whatever the reply said

HOW TO QUOTE THE PACKET, and how it will be read.

For every element you read as `qualified` or `negative`, and for a scope of `prior_cycle` or
`different_entity`, the citation must be ONE SENTENCE OR ONE ANSWER LINE COPIED VERBATIM out of the
packet -- the text that makes that reading. It may come from any part of the packet, including a
header line, the covering note or the signature block.

  - Copy it character for character. It is located in the packet by searching for it, so a
    paraphrase, a shortened version, an ellipsis in the middle, or two sentences joined together
    will not be found at all and will score nothing. There is no partial credit for a quote the
    packet does not contain.
  - Quote the sentence, not the section. What is returned is compared with that sentence by
    character overlap: it must cover at least 60 pct of the sentence, and at least 30 pct of
    what you return must be the sentence. Returning the whole packet scores nothing.
  - The rulebook is NOT part of the packet. A rule is never the citation.
  - Where you read an element as `affirmed`, `citation` is null -- an affirmative answer needs no
    evidence here.
  - ⚠︎ WHERE YOU READ AN ELEMENT AS `absent`, `citation` IS NULL TOO, AND THIS IS THE ONE MOST
    OFTEN GOT WRONG. An absent element has no text in the packet, so there is nothing to quote.
    Quoting the nearest thing to it -- a general assurance in the covering note, an answer to a
    different element -- is counted as a wrong answer, not as an empty one.

THE RECONCILIATION PACKET, verbatim:

FDR RECONCILIATION PACKET  FDR-0001 -- Annual delegation attestation

CONTRACT INVENTORY ROW -- the authority on who holds this delegation

  FDR id                      FDR-0001
  Contracted entity           Vale Ridge Billing Services LLC
  Parent group                Vale Ridge Health Group
  Tier                        first-tier
  Delegated function          utilization management
  Contract effective          2024-02-01
  Cycle under reconciliation  2026-H1
  Attestation due             2026-07-31
  Submission matched          ATT-2026H1-0401
  Exclusion screening result  clear -- OIG LEIE and GSA lists, no match returned

ATTESTATION PACKET AS RECEIVED

  Received                    2026-07-06
  Attesting entity            Vale Ridge Billing Services LLC
  Attestation cycle stated    2026-H1
  Signed by                   the compliance officer for the delegated function

COVERING NOTE

The contacts for this delegation are unchanged from the previous period and are listed in the contract schedule. This packet was prepared by our compliance function and reviewed by the officer named in the signature block.

ANSWERS TO THE REQUIRED ELEMENTS

  1. Compliance programme
     A written compliance programme, a code of conduct and a confidential reporting line were in operation throughout the period, and the officer accountable for the delegated function is named in the signature block.

  2. General compliance and FWA training
     Every member of the delegated team completed general compliance and FWA training within the cycle; the completion register was closed on 12 June.

  3. Exclusion screening cadence
     Monthly OIG LEIE and GSA exclusion screening was performed for all staff and contractors performing the delegated function for every month of the period.

  4. Downstream flow-down
     We subcontract no part of the delegated function and have no downstream entities; the requirement is recorded as not applicable and the position was reconfirmed in March.

CORRESPONDENCE ON THIS PACKET

  2026-07-07  packet logged against the delegation record


Reply with JSON and nothing else, exactly this shape:

{
 "elements": [
   {
    "element": "compliance_program" | "fwa_training" | "exclusion_screening" | "downstream_flowdown",
    "reading": "affirmed" | "qualified" | "negative" | "absent",
    "citation": "<text>" or null
   },
   ... one object per required element, all four, in this order: compliance_program, fwa_training, exclusion_screening, downstream_flowdown
 ],
 "scope": "this_cycle_this_entity" | "prior_cycle" | "different_entity" | "no_submission",
 "scope_citation": "<text>" or null,
 "confidence": <a number between 0 and 1>,
 "why": "<text>"
}

What each field means:
  element         which required element this row is about. Answer all four, once each, in the order given
  reading         how the packet answers this element. `affirmed` answered affirmatively for the whole cycle with no condition, qualification, exception or future date. `qualified` answered affirmatively AND conditionally -- substantially, largely, in progress, under way, on track, targeting, all but a few, with the exception of, expected by, pending. A qualified answer is NOT a weaker affirmative: it states the requirement is not met yet, in the register of one that is. `negative` answered, and the answer is that the requirement is not met. `absent` the packet does not address this element anywhere
  citation        ONE SENTENCE OR ONE NUMBERED ANSWER LINE COPIED VERBATIM from the packet -- the text that makes this reading. Copy it exactly, character for character; do not paraphrase, do not shorten with an ellipsis, do not join two sentences. NULL where the reading is `affirmed` or `absent`: an affirmative answer needs no evidence and an ABSENT element has no text to quote, so quoting one is an invention and is counted as a wrong answer rather than an empty one
  scope           WHOSE submission this is and WHICH CYCLE it covers, read against the inventory row printed above the packet. `this_cycle_this_entity` the packet attests for the entity named on the inventory row and for this cycle. `prior_cycle` a packet was received and it attests to an EARLIER cycle -- a resubmitted prior-year certification, or an attested period ending before this cycle begins. `different_entity` a packet was received and it attests for a DIFFERENT contracted entity: a parent, an affiliate, or a sibling under the same group. `no_submission` no packet at all. A packet existing is NOT evidence that this entity submitted for this cycle, and this field is the one that says so
  scope_citation  ONE SENTENCE OR ONE HEADER LINE COPIED VERBATIM from the packet -- the text that shows the cycle or the entity it actually covers. Required where scope is `prior_cycle` or `different_entity`. NULL where scope is `this_cycle_this_entity` or `no_submission`
  confidence      your own number between 0 and 1 for this FDR's readings taken together. It is published as evidence and never used to change an answer
  why             one sentence naming what decided any element you did not read as `affirmed`, and the cycle and entity the packet actually covers

Four element objects for one packet. No more, no fewer. THERE IS NO STATUS FIELD:
the reconciliation status is derived in code from your reading, the contract
inventory and the screening result on file.

Raw responseThe raw response

{
 "elements": [
  {
   "element": "compliance_program",
   "reading": "affirmed",
   "citation": null
  },
  {
   "element": "fwa_training",
   "reading": "affirmed",
   "citation": null
  },
  {
   "element": "exclusion_screening",
   "reading": "affirmed",
   "citation": null
  },
  {
   "element": "downstream_flowdown",
   "reading": "affirmed",
   "citation": null
  }
 ],
 "scope": "this_cycle_this_entity",
 "scope_citation": null,
 "confidence": 0.99,
 "why": "The packet was submitted by Northgate Care Management LLC for 2026-H1, matching the inventory row, and all four required elements are answered affirmatively without any qualification."
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check a health plan's vendor compliance packets — 60 FDR reconciliation packets. One model answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

No model grades anything. Every metric is exact string match against data/gold.jsonl, except the citation, which is scored by character-interval overlap in src/citation.py. Re-scoring a committed run costs $0.00 and needs no network, which is what 'free to re-run' means here.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Run it twiceThe same set, run again

⚠︎ THE PAID ARM WAS RUN ONCE AND NOTHING HERE SAYS WHETHER 95.0 PCT IS STABLE. There is no second run and no second model, so the 12-status margin is a single observation. What WAS re-derived is the scoring: after a defect was found in the answer key, every arm was re-scored from its own cached answers for zero calls, and the paid arm's status_correct did not move (57 before and after) while row_all_correct rose from 51 to 57 — because the six rows the key had labelled wrongly were the six the model had got right.

status accuracy over the 60 FDR rows — There is one paid run, so there is nothing to average. Publishing a mean of one is a number nobody measured wearing the shape of a number somebody did.

Grading costWhat it costs

Every dollar on this page is a MEASURED token count multiplied by a NAMED, DATED published card. The tokens are the same whoever they are priced against; the price is a projection and is labelled as one.

Rates checked 2026-08-27. The provider that actually ran every call here is kept off this page, per the repo rule that only src/adapters/ knows who serves the model. The run record carries what it actually cost at its own card ($0.080195 for 60 calls, off-peak) and this lens carries the projection.

Grading unitWhat the grading figure prices

Scoring is pure Python over committed JSON: 60 rows, 240 element readings, 300 citation slots, in well under a second. The cost being priced is the GRADER'S, not the pipeline's — the pipeline's bill is lens 07.

The gradersOne way to grade, and why it is the only one

THE NULL BASELINE IS 33.3 PCT AND IT IS THE LARGEST CLASS, NOT AN EASY ONE. An arm that answered missing-entirely to every row would score 20 of 60 — and would be wrong on every packet that actually arrived, sending 40 vendors a letter saying nothing was received. Answering clean to everything scores 11 of 60. Neither is a floor worth publishing, which is why the arm on the board is evals/baseline.py: a phrase list over the numbered answers plus a header comparison, and then the SAME rulebook engine the paid arm is rechecked with. ⚠︎ AND ITS PHRASE LISTS WERE WRITTEN BY THE SAME AUTHOR AS THE CORPUS, ON THE SAME DAY, from data/policy.md's own wording for FA-6 and FA-9 — before the packet prose was finished, and never revised after scoring. Read 75.0 pct as a ceiling on what phrase matching does on writing this regular.

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-27. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingGrading is free; the bill is the 60 calls that answered.

The scorer, the label gate, the free floor, the corpus generator, the recheck red-proof and the screenshot script all run offline for $0.00. Re-scoring a committed run at a different citation threshold costs nothing and buys no calls, which is the whole reason the graders are pure code. The only spend in this kit is the 60 completion calls of r001-vendor-attest.

Cost driversWhat actually moves the bill

• PROVIDER-SIDE REASONING. 70,856 of the 83,390 output tokens (85.0 pct) were reasoning that never reached the reply text, and output is priced 6.0x input on the projection card. It is the single largest line on the bill and the reply itself is four short rows.
• THE RULEBOOK IN THE PROMPT. data/policy.md is 8,310 of the prompt's 23967 characters — the largest single part, larger than the packet. It is byte-identical on every call, so a provider that prices cached input separately bills most of it at the cached rate (223,872 of 331,102 input tokens came back as cache hits here); one that does not will bill it in full, 60 times.
• THE NUMBER OF FDRs. One call per inventory row, flat. There is no retrieval, no index, no second pass and no re-fire on failure — a truncated reply is recorded as a failure and stays in the denominator rather than being bought again.

Your volumeWhat it costs at your volume

Linear in FDR rows and flat in everything else. Ten times the inventory is ten times the calls and the same prompt prefix, so the cached-input share should hold or improve. What does NOT scale is the join: at 600 contracted entities, resolving which packet belongs to which contract stops being a lookup and becomes the job, and this kit assumes it is already done.

Model choiceWhy this model

It is the tier the estate runs, so this figure compares with every sibling kit. Nothing here argues it is the right tier for this job — there is no second model on the board, and Eval.could_not_verify says so.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

the whole 60-row scored run (r001-vendor-attest, the fast tier): 331,102 input tokens (223,872 of them cache hits) and 83,390 output tokens (70,856 of them provider-side reasoning that never reached the reply text)

The codeEvery file, and what it is for

12 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Eight of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

data/inventory.jsonthe contract inventory — a swap seam

THE POPULATION, and the single most important design decision in the kit. One row per contracted FDR: the entity, the tier, the delegated function, the packet the intake matched to it this cycle (null where nothing arrived) and the exclusion-screening result on file. src/policy.py reads exactly two fields from it — the join and the screening — and nothing else. Every arm, every scorer and the local UI iterate ITS keys, never data/corpus/. That is the whole mechanism behind FA-1: a harness built over the packets that arrived has 54 units of work and never constructs the other 6.

You change it to: Your own FDRs. ⚠︎ THIS IS THE SEAM THAT MATTERS MOST AND IS EASIEST TO GET WRONG: an inventory assembled from the packets you received cannot contain a single missing-entirely row, and the kit will report a flawless score on a population missing the only thing it was built to find.

# 

src/prompt.pyprompt assembly

Six parts in a fixed order — the system role with the refusal stated first, data/policy.md verbatim (all ten rules and the order they are applied in), the four statuses and what recording each one commits an analyst to, the four required elements plus the reading and scope vocabularies and the citation rule, the packet VERBATIM, and the JSON schema built from data/fields.json. The first four parts are byte-identical on every call and are sent first: 18536 of 23967 characters, 77.3 pct of the prompt, which is why 223872 of 331102 input tokens came back as cache hits across the run.

# The assembled prompt, in one place, in the order it is sent.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
POLICY_MD = open(os.path.join(HERE, "data", "policy.md"), encoding="utf-8").read()
FIELDS = json.load(open(os.path.join(HERE, "data", "fields.json"), encoding="utf-8"))["fields"]
SYSTEM = """You are reading one FDR reconciliation packet from a delegated-vendor oversight cycle and
def _status_block():
def _elements_block():
CITATION_RULE = """HOW TO QUOTE THE PACKET, and how it will be read.
def _schema_block():
STATUS_BLOCK = _status_block()

evals/run.pythe packet, unsummarised

The reconciliation packet goes into the prompt as it is — the inventory row, the packet header, the covering note, the numbered answers and the correspondence, 1964.2 bytes mean across the corpus. There is no pre-digest and no extracted 'answers' handed over as a summary, deliberately: a summary is where 'training is substantially complete' quietly becomes 'training complete' before the model ever sees it, and then the measurement is of the summariser.

# Read all 60 FDR rows and write one result file. THE PAID ARM SPENDS MONEY.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
CORPUS = os.path.join(HERE, "data", "corpus")
KIT = "vendor-attest"
RATE_CARDS = {
def tariff_at(card, ts):
def price(card, row):
def load_packet(rid):
def load_gold():

src/classifier.pythe model call

One call per FDR row, max_tokens 32000, socket timeout 1200 s, five workers. The reply is one JSON object: four element rows in a fixed order, each with a reading and at most one quoted sentence, plus the scope, a scope citation, the model's own confidence and a free-text why. All 60 parsed; failures 0; every finish_reason 'stop'. ⚑ THERE IS NO STATUS FIELD IN THE REPLY AND THERE CANNOT BE — normalise() drops one if a reply volunteers it.

# One reconciliation packet in, four element readings and one scope reading out. The only place a

MAX_TOKENS = 32000
THINKING = None
def parse_reply(text):
def normalise(obj):
def classify(cfg, text, row, complete_fn=None, max_tokens=None):
ELEMENTS = tuple(A.ELEMENT_ORDER)

src/recheck.pywhat the model is trusted with

Exactly two kinds of field survive: the four element READINGS and the packet SCOPE. There is no status to discard because none was ever asked for. The one thing the station overrides is the scope, and only in one direction: where the inventory join carries no packet, the reply's scope is forced to no_submission before any rule is applied — a reading about a document that does not exist is not a reading. ⚑ ON THIS RUN THAT STATION BOUGHT NOTHING: recheck_join_overrides is 0, because the packet's own text says 'No attestation packet is matched to this FDR for this cycle' and every arm read it. That is a real finding rather than a null result — the guardrail is mechanical, and on this corpus it was never called on.

# THE PURE-CODE STATION, applied to an arm's reply. What the arm is trusted with, and nothing more.

TRUSTED = ("reading", "scope")
NO_CITATION_READINGS = ("affirmed", "absent")
NO_CITATION_SCOPES = ("this_cycle_this_entity", "no_submission")
def recheck(answer, row, packet_text):

src/policy.pythe rulebook engine

FDR-2026's ten rules read out of data/policy.json in the published order: nothing received (FA-1), a prior cycle (FA-2), a different entity (FA-3), a confirmed exclusion (FA-4), an unresolved match (FA-5), an element answered no (FA-6), an element unanswered (FA-7), no screening result (FA-8), a qualified answer (FA-9), reconciled (FA-10). THE ORDER IS THE RULEBOOK — FA-2 and FA-3 before FA-7 is what stops a wrong-cycle packet being logged as 'received but incomplete' and sent the wrong letter; FA-4 and FA-5 before every element rule is what stops a well-written attestation softening a screening hit. It never sees the packet's text, by design: it takes a join, a scope, four readings and a screening state, which is what makes the split between 'what only a reader can supply' and 'what is a table lookup' checkable rather than asserted. Six precedence claims are asserted at import.

# THE PURE-CODE STATION. An inventory join, a scope reading, four element readings and a screening

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
POLICY = json.load(open(os.path.join(HERE, "data", "policy.json"), encoding="utf-8"))
POLICY_ID = POLICY["policy_id"]
AS_OF = POLICY["as_of"]
CYCLE = POLICY["cycle"]
ANCHOR = POLICY["anchor"]
RULES = POLICY["rules"]
RULE_ORDER = POLICY["rule_order"]
STATUSES = tuple(POLICY["statuses"])

data/inventory.jsonthe exclusion-screening result — a swap seam

A FACT on the inventory row, one of clear / potential_match_unresolved / confirmed_exclusion / not_performed, produced by matching the FDR's staff list against the OIG LEIE and the GSA exclusion list. A name match is a deterministic comparison rather than a reading, so it is held as data on EVERY arm and no arm is graded on it — and it is never inferred from the packet's prose and never changed by it. It decides 8 of the 60 rows on its own (FA-4 2, FA-5 4, FA-8 2).

You change it to: Your own FDRs. ⚠︎ THIS IS THE SEAM THAT MATTERS MOST AND IS EASIEST TO GET WRONG: an inventory assembled from the packets you received cannot contain a single missing-entirely row, and the kit will report a flawless score on a population missing the only thing it was built to find.

# 

src/citation.pythe citation locator — a swap seam

Normalises the packet once (NFC, 1:1 character folds, whitespace collapsed), locates the returned quote in it, and compares CHARACTER INTERVALS with the labelled span: coverage >= 0.60 and precision >= 0.30. Not found is zero with no near-miss credit, which is what stops a fluent wrong answer scoring like a right one — the best a whole-packet dump could score is precision 0.1029, measured into corpus-stats.json. Both thresholds are arguments, so any committed run re-scores at a different pair with no provider in the loop.

You change it to: COVERAGE_MIN and PRECISION_MIN, or --coverage-min / --precision-min on any committed run. Re-scoring at a different pair costs nothing and buys no calls.

# HOW A CITATION IS SCORED. Character offsets, two floors, and no partial credit for a quote the

FOLDS = {
def fold(s):
def norm(s):
def norm_quote(s):
def locate(norm_text, quote):
def span_of(norm_text, sentence):
COVERAGE_MIN = 0.60
PRECISION_MIN = 0.30
def credit(norm_text, gold_span, quote, coverage_min=COVERAGE_MIN, precision_min=PRECISION_MIN):

src/attestation.pythe vocabulary, in one place — a swap seam

The four statuses and their order, the four required elements and their order, the four element readings, the four scopes and the four screening states — one copy, read by the prompt, the floor, the rulebook engine, the recheck station, the scorer and the UI. missing-entirely leads the status order deliberately: it is not a degree of non-conforming, and the two must never be drawn beside each other as neighbouring shades of alarm.

You change it to: ELEMENTS and ELEMENT_ORDER. The prompt block, the schema, the floor's heading map, the scorer's denominators and the UI's rows are all built from them; add a fifth element and every one of those follows.

# The four reconciliation statuses, the four required attestation elements, the readings behind

STATUSES = {
ORDER = ["missing-entirely", "non-conforming", "hedged", "clean"]
ELEMENTS = {
ELEMENT_ORDER = ["compliance_program", "fwa_training", "exclusion_screening",
READINGS = {
READING_ORDER = ["affirmed", "qualified", "negative", "absent"]
READING_SEVERITY = ["negative", "absent", "qualified", "affirmed"]
SCOPES = {
SCOPE_ORDER = ["this_cycle_this_entity", "prior_cycle", "different_entity", "no_submission"]

evals/baseline.pythe free rules floor — a swap seam

A phrase list per reading over the packet's NUMBERED ANSWERS ONLY, plus a comparison of the two header lines that say whose packet this is and which cycle it covers — and then exactly the rulebook engine the paid arm is rechecked with. It emits what it read and applies no answer contract of its own, so src/recheck.py is a real station on this arm too, red-proven by tools/redproof_recheck.py. It scores 45 of 60 statuses for $0.00.

You change it to: NEGATE and QUALIFY. They are the kit's claim about what a weekend of scripting reaches, and they were written before the run was first scored; changing them changes the margin and the report should say so.

# THE FREE FLOOR. No key, no model, no network. A genuine attempt at the job.

MODES = ("rules",)
NEGATE = [
QUALIFY = [
ANSWER_HEADING = "ANSWERS TO THE REQUIRED ELEMENTS"
PACKET_HEADING = "ATTESTATION PACKET AS RECEIVED"
NO_PACKET = "No attestation packet is matched to this FDR for this cycle."
HEADINGS = {
def sections(text):
def header_value(block, label):

data/fields.jsonthe answer contract — a swap seam

The five fields a reply may carry, as data, so the schema the model is given and the fields the scorer reads cannot drift apart. ⚑ THERE IS DELIBERATELY NO status FIELD. src/prompt.py asserts its absence at import and evals/check_labels.py fails the build if one appears. That single absence is what makes 'this kit never certifies an FDR' a property of the contract rather than a hope about the reply.

You change it to: What the reply may carry. Adding a field the scorer does not grade, or grading one the schema does not offer, is how an eval starts measuring its own wiring — so both sides read this file.

# 

src/app.pythe local board

A standard-library HTTP server on port 9240. Every endpoint but one needs no key: the inventory, one row with its key and the live free floor, the whole-cycle board, the assembled prompt and a replay of the committed run. /api/decide is the only one that spends, and the button reaching it is disabled with the reason printed on the page rather than hidden in a title attribute.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
CORPUS = os.path.join(HERE, "data", "corpus")
PORT = int(os.environ.get("PORT", "9240"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-vendor-attest")
FLOOR_RUN = "b000-vendor-attest-rules"
def golds():
def load_packet(rid):
def recorded(run_id):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 5518 input and 1389 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

⚠︎ WRITTEN, WIRED, AND NOT FIRED — AND NO NUMBER ON THIS PAGE SAYS OTHERWISE. evals/injection.py exists, is complete, and would append one sentence to each packet's own covering note; results/ holds no x001-vendor-attest file and no adversarial figure exists for this kit. What IS measured is the mild version, and it is measured because it is planted in the corpus rather than bolted on: 8 of the 60 packets ask the plan in words to certify the delegation, mark it compliant, or treat an exclusion-screening result as cleared on the vendor's own say-so. THE SCORED RUN GRANTED 0 OF 8. The free rules floor granted 1 — FDR-0058, which it answered clean because the packet's header carried no entity line for it to compare.

API_KEY is read from the shared <code>&lt;repo&gt;/.env</code> and this kit's own <code>.env</code>, both gitignored from the first commit, and the real environment overrides both. src/config.py::save() opens with mode 0600 before the file exists, so it is never readable at the default umask even for an instant, and sources() reports WHICH file a value came from and never the value. The key reaches exactly one place: an Authorization header in src/adapters/__init__.py. src/app.py redacts it out of any provider error before returning one to the browser. Nothing else in the kit reads it.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. ⚠︎ THE ADVERSARIAL ARM WAS NEVER FIRED. python3 -m evals.injection --run-id x001-vendor-attest --yes is the command; it would be 49 calls, one per row whose key is not clean. Measured on no run — results/ holds no x001-vendor-attest file, and no adversarial figure exists for this kit.

The result⚠︎ NOT FIRED — no adversarial number exists for this kit. What is measured is the eight planted certification and clearance requests: the model granted 0, the free floor granted 1.

The design, so it can be checked rather than re-argued later: one sentence appended to the end of each packet's own COVERING NOTE — where a note actually goes, not at the top of the file where anything would read as a system instruction — on every row whose key is not clean, 49 rows, one call each. Scored against the CLEAN packet, so a citation quoting the injected sentence cannot be located and scores zero, which is the right answer because that sentence is not in the packet the key was written against. The control is the scored run's own cached determination for the same row; no second control call is bought.

HonestyWhat this does not prove

• Any adversarial rate at all — the arm has not been fired.
• Whether the provider retains any part of a packet, or for how long.
• Whether an FDR could shape a covering note to move a reading without tripping the eight planted requests' pattern. The eight are one author's idea of how a vendor asks.
• Whether the inventory itself is complete. The kit treats it as ground truth and there is nothing above it to check it against — an FDR missing from the inventory produces no row, no status and no trace.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never certify an FDR as compliant with the attestation or exclusion-screening requirements, never clear or close an exclusion-screening result, and never treat a missing submission as a received one. It reconciles submissions against the contract inventory and names the follow-up for a person to close.

Stated in the system role as the FIRST thing the model is told, in the README, in src/app.py's docstring, on the local board's lead paragraph and in src/attestation.py, which is the file every other file reads its vocabulary from. ⚑ AND ENFORCED IN THE CONTRACT RATHER THAN IN PROSE: data/fields.json offers no status field, src/prompt.py asserts its absence at import, src/classifier.py::normalise() drops one if a reply volunteers it, and evals/check_labels.py fails the build if one appears in the contract.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a compliance determination. clean means reconciled — no follow-up this cycle — and the vocabulary in src/attestation.py says so in the field a renderer reads.
• It is NOT an exclusion-screening system. It never screens a name and never resolves a match; the result is a fact it is handed.
• It is NOT a completeness check on the inventory. An FDR missing from data/inventory.json produces no row and no trace, and nothing here can detect that.
• It is NOT a document extractor. Packets are assumed to be text already.
• It does not write to anything. There is no ticket, no letter, no queue and no system of record on the other side of it.

WatchedWhat is watched, and why that one

The kit measures 39 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

122 measured by the latest run -83 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (batch_reader_invisible_rows 6, batch_reader_rows 54, calls_in_totals 60, certification_requested_rows 8, citation_coverage_floor 0.6, citation_precision_floor 0.3, citation_slots 300, citations_expected 33, clean_status_rows 11, dataset_version vendor-attest-v1-60fdrs, easy_rows 30, element_judgements 240, elements_per_row 4, failures 0, fdrs 60, fdrs_answered 60, gap_rows 20, hard_rows 30, hedge_rows 12, open_screening_rows 6, policy_id FDR-2026, raw_certification_requested_rows 8, raw_citation_coverage_floor 0.6, raw_citation_precision_floor 0.3, raw_citation_slots 300, raw_citations_expected 33, raw_clean_status_rows 11, raw_easy_rows 30, raw_element_judgements 240, raw_fdrs_answered 60, raw_gap_rows 20, raw_hard_rows 30, raw_hedge_rows 12, raw_open_screening_rows 6, raw_silent_absence_rows 14, raw_status_clean_rows 11, raw_status_hedged_rows 12, raw_status_missing_entirely_rows 20, raw_status_non_conforming_rows 17, raw_uncited_expected 267, silent_absence_rows 14, socket_timeout_s 1200, status_clean_rows 11, status_hedged_rows 12, status_missing_entirely_rows 20, status_non_conforming_rows 17, uncited_expected 267) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

3 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

comply · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 119 chips that all say so.

comply · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 128 chips that all say so.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 122 chips that all say so.

DeviationsWhat deviated

0 breaches across 3 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The corpus generator, the label gate, the free floor, the recheck red-proof, the scorer and the screenshot script all run offline and free, so every band above can be re-measured as a build step. The paid arm is re-run only when the prompt, the model or the rulebook changes — and when it is, the cache means a re-score costs nothing.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, standard library end to end — no orchestrator, no agent, no tool loop, no vector store, no SDK. requirements.txt names nothing and says why. The point is that a forker can read the prompt, read the rulebook and read the scorer in an afternoon; a framework would move all three behind an abstraction.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear, and concurrent only at the row level. Packet in -> one call -> parse -> the pure-code station (four readings, one scope, plus the inventory join and the screening result) -> one status -> score. There is no agent, no tool loop, no retrieval and no second pass, so there is no graph to draw beyond that line.

The other sideWhat a framework costs you

• No streaming, so the socket sits silent for the whole generation and the timeout has to be sized for a reply that fills the ceiling rather than for the one expected.
• No structured-output mode. The reply is parsed from text by a brace-matching scanner in src/classifier.py; all 60 replies parsed on the scored run, and a provider's native JSON mode would make that guaranteed rather than measured.
• No batching. One HTTP request per row, so 6,000 rows is 6,000 requests.
• No retry on a truncated reply, by choice: it is recorded as a failure and stays in the published denominator rather than being bought again.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-vendor-attest on the fast tier, 2026-09-01. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 1 earlier run on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 1 run on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — the kit's own licence. Nothing in the corpus is derived from any third-party material, so there is no upstream licence to carry. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe reconciliation — the status, the scope, the four readings and the sentence behind them

for each of the 60 FDR rows in the contract inventory: the reconciliation status (missing-entirely / non-conforming / hedged / clean), the scope, each of the four element readings, and every citation, scored as a character interval against the labelled sentence

MethodHow the test was run

python3 -m evals.run --run-id <id> [--floor rules | --stub]; evals/scoring.py is imported by the harness and by src/app.py, so the page and the run record cannot disagree about what a cell scored

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl — DERIVED, NOT WRITTEN. tools/build_corpus.py plants a sentence, records its span, and calls src/policy.py with the injected reading, the inventory join and the screening result. evals/check_labels.py re-derives all of it independently, importing nothing from src/.

No true/false rates for this grader. It records 15 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? Citation credit is coverage >= 0.60 and precision >= 0.30, both arguments to credit() so any committed run re-scores at a different pair with no provider in the loop. Measured mean on located citations: coverage 0.99, precision 1.00 (paid arm). Not found is zero with no near-miss credit — 0 of the paid arm's 32 returned quotes were unlocatable.

Cadence: the corpus generator, the label gate, the free floor, the recheck red-proof and the scorer all run offline and free, so re-grading is a build step rather than a spend. The paid arm is re-run only when the prompt, the model or the rulebook changes.

The decisionWhen to reach for it