The business caseThe problem this solves

A group health plan has to file a Section 111 record for every member it covers who is also a Medicare beneficiary — 42 U.S.C. 1395y(b)(7) makes it mandatory — and CMS sends back a response file saying which records it refused. Somebody then works the refusals: for each one, why was it refused, and what has to change before the next quarterly submission. Today that is an analyst opening a packet, comparing the staged record against the plan's own eligibility snapshot field by field, reading the response code, reading whatever the last person wrote in the notes, and deciding. The per-packet reconciliation an analyst does by hand on a Section 111 response file — field-by-field comparison, response-code lookup, note reading, and the decision that follows. It does not replace the analyst: every answer is a proposal and nothing is submitted or written.

The inputThe actual reconciliation packets

The corpus is 60 reconciliation packets, 0.12 MB (txt 60). Because the thing being measured is a distinction the source data cannot contain as ground truth. CMS returns the SAME refusal for a keying error and for a different person; whether a given mismatch is one or the other is only visible inside the plan's own packet, and no real corpus carries that as a label. So the facts are injected and the answer key is DERIVED from them by src/policy.py — and the perturbation is applied to the data, not described in prose: a packet labelled a keying error really does disagree on exactly one identifier, and a wrong-person packet really does carry another individual's values. evals/check_labels.py re-parses both blocks and fails if that stops being true, which is what makes the free floor's field diff a genuine arm rather than a strawman.

SECTION 111 RECONCILIATION PACKET MSP-0001 -- Response file, record accepted

PACKET FACTS
  Responsible reporting entity      Tarnsyde Municipal Employees Plan
  Reporting cycle                   2026Q2, cut-off 2026-06-30
  Packet opened                     2026-07-27
  Register read as at               2026-09-01

STAGED OUTBOUND RECORD -- what the plan is about to send

  Medicare Beneficiary Identifier   5MY5-G66-NK15
  Social Security Number            943-42-8154
  Surname                           Penhallow
  First name                        Rosa
  Date of birth                     1959-11-18
  Coverage effective date           2021-07-01
  Coverage termination date         2026-01-20
  Relationship to subscriber        dependent of subscriber

PLAN ELIGIBILITY SNAPSHOT -- what the plan's own membership system holds

  Medicare Beneficiary Identifier   5MY5-G66-NK15
  Social Security Number            943-42-8154
  Surname                           Penhallow
  First name                        Rosa
  Date of birth                     1959-11-18
  Coverage effective date           2021-07-01
  Coverage termination date         2026-01-20
  Relationship to subscriber        dependent of subscriber

CMS RESPONSE FILE, PRIOR CYCLE

  no record for this member was included in the prior cycle

ANALYST NOTES AND CORRESPONDENCE

  The packet was assigned to the reconciliation queue and has not been reassigned.
  The packet was opened when the response file was loaded and has been added to twice since.
  Membership was asked to confirm the enrolment and replied the same day.

CORRESPONDENCE LOG

  2026-08-04  response file loaded and split by disposition
  2026-08-09  packet opened and assigned to the reconciliation queue

The outcomeWhat a good result looks like

Four answers a compliance analyst approves instead of re-reading the packet: the cause, the correction, the data element at fault, and the line that shows it. 58 of 60 packets came back with all four right at $0.001549 per packet.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• A response file whose refusals are the ordinary kind — a legacy identifier, a coverage date that contradicts the snapshot, an entitlement CMS does not hold — and whose packets carry the plan's own eligibility snapshot beside the staged record — the free rules floor alone
  Five results are a TIE on this corpus and they are not small: demographic_typo 8/8 both, identifier_invalid 5/5 both, coverage_dates 5/5 both, not_entitled 5/5 both, and the whole clean control group 9/9 both. Every register-driven answer is free too — MSP-6 4/4 and MSP-7 5/5 on both arms — because they come from data/records.json and no reading is involved. On this slice a field diff and a code lookup are the whole job.
• A response file where identifier mismatches are common and somebody has to decide, packet by packet, whether each one is a keying error or a different member — the paid call, and read wrong_person_read_as_correctable rather than the aggregate
  This is the only slice where the two arms separate on the thing that matters. The floor held 11 of 14 wrong-person packets and read 3 as correctable; the paid call held 14 of 14 and read 0 as correctable. The floor's three misses are all the sharp form — exactly one identifier out, with the second member named in wording its phrase list does not carry — and each of the three would have written another member's data into a federal filing at $0.00.
• A response file whose codes are quoted in analyst notes about OTHER records in the same submission, or about errors already fixed — the paid call
  The 5 decoy packets are exactly that, and a grep-based code lookup cannot tell whose record a code is about. The floor over-corrected 4 of them; the paid call over-corrected 0.

And where nothing here is good enough:

• Packets where nobody wrote an analyst note stating the field comparison — neither, on this evidence
  NOT MEASURED. Every packet in this corpus carries a note that states the comparison in words, and reading it is easier than doing it. A notes-stripped run would separate the two arms on the thing this kit claims to measure; it costs about $0.09 and it was not fired.

At a glanceHow the whole thing runs

Run once, for real, on 2026-09-01. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-09-01 — r001-secondary-payer. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A clean checkout with no key configured renders the whole board, scores the free floor offline and replays the committed run for $0.00 — observed on this machine, with API_KEY blanked, by the screenshot script that took all four frames in docs/shots/. requirements.txt names no package; the kit is Python standard library end to end and pip install pulls nothing.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END TO END for one packet on the scored run r001-secondary-payer: prompt assembly, one completion call, the JSON parse, the pure-code recheck and the citation locate. Measured over 60 packets at 5 workers. The p95 is 4.3x the p50 and the tail is provider-side reasoning, not network — 94,981 of 102,628 output tokens on this run never reached the reply text.

Current processWhat it replaces

The per-packet reconciliation an analyst does by hand on a Section 111 response file — field-by-field comparison, response-code lookup, note reading, and the decision that follows. It does not replace the analyst: every answer is a proposal and nothing is submitted or written.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one reconciliation packet is

No split, and that is a decision rather than an omission. Every packet is in the scored set; there is no training, no few-shot selection and nothing fitted to any subset, so holding packets back would only shrink the denominator. The corpus is stratified instead: 13 planned cases, 9 clean and 51 hard, with every class a counted denominator on the report.

SetupWhat the setup figure measured

THERE IS NO INDEX AND THE ZEROES ARE LITERAL, NOT UNMEASURED. Nothing is embedded, chunked, retrieved or persisted between calls: one packet goes whole into one call. The two files a reader might mistake for an index are neither — data/records.json is the plan's coordination-of-benefits register, one row per packet, read as data by src/policy.py and never searched; data/gold.jsonl is the answer key and is not in the answering path at all. What the free floor computes in memory each run is the field diff between the two identifier blocks inside each packet, which costs nothing and is the only pre-model reduction in the kit. Regenerating the whole corpus and its key with tools/build_corpus.py takes under a second and makes no call.

LicenceLicence

MIT, and there is no third-party material in it to licence. Nothing here is derived from any dataset, document or system: the packets are generated from phrase pools written for this kit.

Bring your ownBring your own reconciliation packets

Drop your own packets as .txt into data/corpus/, add one row per packet to data/records.json (id, title, case, hard, plan, cycle, opened_on, and a register dict giving cob_status one of current / stale / not_established), and write data/gold.jsonl with the cause, the correction, the rule, the data element, the citation and its span. The span is computable rather than counted by hand: src/citation.py::span_of(norm(text), line). Replace data/policy.json and data/policy.md with your own wording — the rule ids, their order and their outcomes are read, never hard-coded — and add a rule for any condition the eight here do not cover. Then run python3 -m evals.check_labels: it refuses a key that does not follow from its own packet, and it will tell you which row.

⚠︎ And what stops being true when you do: ⚠︎ EVERY NUMBER ON THIS PAGE STOPS BEING TRUE THE MOMENT YOU DO. The measured result rests on packets whose analyst notes STATE the field comparison in words; on packets where nobody wrote that note, this is an untested pipeline. And the disclosure question is yours, not the kit's: a reconciliation packet carries a member's Medicare identifier, Social Security Number, name and date of birth, and the WHOLE PACKET goes to a model provider verbatim. Run it against a de-identified extract you are permitted to disclose, or against a local model.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 23,274 characters assembled, 8,742 are vocabularies — 38% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Replayed from the kit's own src/prompt.py::build() for packet MSP-0001, not logged by the run. That is only trustworthy because the replayed decomposition matches what the run recorded: the run's prompt_parts are system 2300, policy 7406, causes 3573, corrections 5169, packet 1954, schema 2797 and its prompt_chars_total is 23199, which is what build() produces for the same packet today. A reader can check that themselves against results/eval-r001-secondary-payer.json.

Full promptThe prompt in full

You are reconciling one Section 111 packet before a group health plan's next quarterly submission to
CMS, and classifying what CMS sent back about it. Your output is the row a compliance analyst
approves -- so that the analyst approves four answers instead of re-reading the packet.

YOU DO NOT SUBMIT ANYTHING AND YOU DO NOT CHANGE ANY MEMBER'S RECORD. You never write to a
coordination-of-benefits record, a Medicare-status field or an outbound file. You produce a
classification and a proposed correction; a person decides whether it happens.

Mandatory reporting here is 42 U.S.C. 1395y(b)(7) -- Section 111 of the Medicare, Medicaid and SCHIP Extension Act of 2007 -- mandatory Medicare Secondary Payer reporting by group health plans.

How to read the packet:

- THE STAGED OUTBOUND RECORD, the PLAN ELIGIBILITY SNAPSHOT, the CMS RESPONSE FILE, the ANALYST
  NOTES AND CORRESPONDENCE and the SUBMISSION HISTORY are all part of the packet and all of them
  count. A packet often shows its cause only in the analyst notes while the response file carries
  nothing but a code.
- COMPARE THE STAGED RECORD WITH THE PLAN'S OWN SNAPSHOT, FIELD BY FIELD. How MANY identifiers
  disagree is the question, not how badly one of them does. That comparison is the whole of the
  mismatch rule below and most of what this packet set is built to test.
- A CMS ERROR CODE IS NOT A CAUSE. The response file says the identifiers did not match a Medicare
  beneficiary; it does not and cannot say whether that is a keying error or a different person. An
  answer that reads the cause off the code alone has not read the packet.
- A code or a phrase does not show a cause just by appearing. A standing note about the file's own
  error counts, a line about a DIFFERENT record in the same submission, or a note recording an
  error that was already fixed all carry the vocabulary and show nothing.
- Answer the four fields once each. Most packets are ordinary: `none` and SUBMIT-AS-STAGED is the
  commonest shape of a correct answer here.
- Apply MSP-2026 as written, INCLUDING THE ORDER ITS RULES ARE APPLIED IN. The first rule whose
  condition holds is the answer.
- Give one confidence between 0 and 1 for this packet's answers taken together.

Reply with JSON and nothing else, in the shape given at the end.

MSP-2026, THE RECONCILIATION POLICY, as approved:

# MSP-2026 — reconciling the Section 111 outbound file, and classifying what CMS sent back

**This policy is invented.** It is not CMS guidance and it is not any plan's reporting policy. What
is real is the obligation it sits under: **42 U.S.C. 1395y(b)(7)** — Section 111 of the Medicare,
Medicaid and SCHIP Extension Act of 2007 — makes reporting mandatory for group health plans, so a
Responsible Reporting Entity sends CMS a file of who it covers and CMS returns a response file
saying which records it took and which it refused. The failure modes below are the ones a response
file really produces. What is invented is the **precedence**, so that it can be checked line by
line rather than argued about.

Nothing here submits a file to CMS, and nothing here changes a member's coordination-of-benefits
record or Medicare status. This produces a reconciliation a compliance analyst approves.

---

## How to read a mismatch

**This is the one judgement in this policy that decides everything else.**

A mismatch is a **KEYING ERROR** where **exactly one identifier disagrees** with the plan's
eligibility snapshot, that disagreement is a keying error in that one field — a transposition, a
dropped or doubled character, an adjacent digit — and **every other identifier agrees**.

A mismatch is a **DIFFERENT PERSON** where **two or more identifiers disagree**, or where one
identifier disagrees and the packet carries something that **establishes the identifiers belong to
somebody else**.

**The response file cannot tell you which it is.** CMS returns the same refusal for both, because
from CMS's side both look identical: the identifiers submitted did not match a Medicare
beneficiary. The difference is only visible inside the packet, against the plan's own snapshot, and
reading it is the whole of what this kit asks a reader for.

---

## The six causes

| cause | what it means |
|---|---|
| `wrong_person_match` | the identifiers resolve to a different individual |
| `demographic_typo` | a keying error in this same member's name or date of birth |
| `identifier_invalid` | the Medicare identifier itself is not a valid current one |
| `coverage_dates` | the coverage effective or termination date is wrong |
| `not_entitled` | CMS holds no Medicare entitlement over the period reported |
| `none` | nothing in the packet points at an error |

## The plan's own register

The **coordination-of-benefits register** is what the plan has done, not what is true of the
member: `current` — re-queried inside this cycle; `stale` — the answer on file predates the cycle
cut-off; `not_established` — the plan has never had one.

---

## The eight rules, in the order they are applied

**The order is the policy.** The same facts under a different order produce different answers.

**MSP-1 — Identifiers that do not resolve to one person are held, never corrected.
→ HOLD-FOR-ANALYST**

Where the identifiers on the staged record resolve to somebody other than the member the record is
about, the record is held for a named compliance analyst. It is not re-keyed, not deleted and not
submitted. **This rule leads the table and outranks everything**, including a clean register and a
record CMS accepted last cycle: correcting a wrong-person match writes one member's name, date of
birth or Medicare identifier onto a second member's coverage report, inside a file that decides who
pays a claim. The correction *is* the harm, so there is no correction.

**MSP-2 — A record CMS holds no entitlement for is deleted, not re-keyed. → DELETE-RECORD**

Where CMS holds no Medicare entitlement for the individual over the period reported, the coverage
was reported in error and a delete transaction removes it. Re-keying it would keep asserting
something untrue for another cycle. This rule sits second because deleting a record whose
identifiers point at the wrong person deletes somebody else's coverage.

**MSP-3 — An identifier that is not a valid current one is replaced. → CORRECT-IDENTIFIER**

Where the Medicare identifier on the staged record is not a well-formed current Medicare
Beneficiary Identifier — a legacy claim number staged after the transition, a wrong length, a
character the format does not allow — it is replaced from the plan's own snapshot and the record is
sent again. This is a fact about the *value* and not about the person, which is why it is separate
from a demographic correction.

**MSP-4 — A keying error in one demographic field is re-keyed. → CORRECT-DEMOGRAPHICS**

Where exactly one demographic identifier disagrees with the plan's own eligibility snapshot, the
disagreement is a keying error in that field, and every other identifier agrees, the field is
re-keyed and the record is sent again. **The condition "every other identifier agrees" is the whole
rule.** Two identifiers disagreeing is not a bigger typo; it is MSP-1, and this rule must never be
reached on that record.

**MSP-5 — A coverage date that contradicts the plan's own snapshot is moved. → CORRECT-DATES**

Where the coverage effective or termination date on the staged record contradicts the plan's
eligibility snapshot, or reports a period overlapping one already accepted, the date is changed to
what the plan holds and the record is sent again.

**MSP-6 — A member with no coordination-of-benefits status established is held.
→ HOLD-FOR-ANALYST**

Where the plan has never established a coordination-of-benefits status for this member and nothing
in the packet names an error, the record is held. The file asserts what the plan knows about a
member's other coverage; a plan that has never asked has nothing to assert, and submitting a
default is an assertion. The four cause rules above outrank this one, because a packet that names
an error has told you something specific and the register has not.

**MSP-7 — A status that predates the cycle cut-off is refreshed before staging.
→ REFRESH-COB-STATUS**

Where the coordination-of-benefits answer on file predates this reporting cycle's cut-off and
nothing in the packet names an error, the plan re-queries and re-stages from the answer it gets.
Nothing about the member changes here. **Reporting a stale status is the failure mode no response
file will ever flag**, because CMS has no way to know the plan's own answer went out of date.

**MSP-8 — A clean packet on a current register is submitted as staged. → SUBMIT-AS-STAGED**

Where nothing in the packet points at an error and the plan's status is current, the record goes in
the next file unchanged. This is the commonest answer on any real cycle. It is not a null result:
it is the plan recording that it looked.

---

## Which corrections name a field

Three corrections edit one named data element and **owe** the element's name, spelled from the
closed list: **CORRECT-IDENTIFIER**, **CORRECT-DEMOGRAPHICS**, **CORRECT-DATES**.

**The other four owe no field and must not name one.** HOLD-FOR-ANALYST, DELETE-RECORD,
REFRESH-COB-STATUS and SUBMIT-AS-STAGED do not edit a data element, and naming a field on one of
them is the first half of an edit the policy forbids — an answer that holds a wrong-person match
and then names the field to fix has already begun doing the thing the hold exists to stop.

It is never inferred. Where the packet does not establish which element is at fault, the answer is
that the field is **NOT STATED**.

THE SIX CAUSES, and what answering each one commits somebody to:

  wrong_person_match   The identifiers resolve to a different individual
      NOTHING IS CORRECTED AND NOTHING IS SUBMITTED. The identifiers on the staged record do not belong to the member the record is about -- two or more of them disagree with the plan's own snapshot, or the packet carries something establishing that they belong to somebody else. Answering this sends the record to a compliance analyst untouched. Answering anything else here re-keys one member's record with another member's data and submits it, which is not a clerical error and is not recoverable by a later cycle.

  demographic_typo     A keying error in this same member's name or date of birth
      SOMEBODY RE-KEYS ONE FIELD AND THE RECORD GOES IN THE NEXT FILE. Exactly one identifier disagrees with the plan's snapshot, the disagreement is a keying error in that field -- a transposition, a dropped character, an adjacent digit -- and every other identifier agrees, so the record is about the person it says it is about.

  identifier_invalid   The Medicare identifier itself is not a valid current one
      THE IDENTIFIER IS REPLACED BEFORE THE RECORD IS SENT AGAIN. The value is not a well-formed current Medicare Beneficiary Identifier -- a legacy claim number staged after the transition, a wrong length, a character the format does not allow. It is a fact about the VALUE, not about the person, and it is why this is a separate cause from a typo.

  coverage_dates       The coverage effective or termination date is wrong
      A DATE MOVES AND THE RECORD IS RE-SENT. The date on the staged record contradicts the plan's own eligibility snapshot, or the period it reports overlaps one already accepted. It is the commonest refusal on a real response file and the cheapest to put right.

  not_entitled         CMS holds no Medicare entitlement over the reported period
      A DELETE TRANSACTION GOES OUT AND THE COVERAGE COMES OFF CMS'S RECORDS. The individual was not a Medicare beneficiary over the dates reported, so the record was reported in error and re-keying it would keep asserting something untrue. Deleting a record that IS true is its own harm in the other direction, which is why this cause has to be established rather than guessed.

  none                 Nothing in the packet points at an error
      NOTHING IS CORRECTED ON THE GROUND OF THE RESPONSE FILE. Either CMS accepted the record, or what the packet carries is about a different record in the same file, or it is a standing note about the file as a whole. What happens next is then decided by the plan's own register, not by this answer.

HOW TO READ A MISMATCH, and it is the one judgement in this policy that decides everything else. A mismatch is a KEYING ERROR where exactly one identifier disagrees with the plan's eligibility snapshot, that disagreement is a keying error in that one field -- a transposition, a dropped or doubled character, an adjacent digit -- and every other identifier agrees. A mismatch is a DIFFERENT PERSON where two or more identifiers disagree, or where one identifier disagrees and the packet carries something that establishes the identifiers belong to somebody else. THE RESPONSE FILE CANNOT TELL YOU WHICH IT IS. CMS returns the same refusal for both, because from CMS's side both look identical: the identifiers submitted did not match a Medicare beneficiary. The difference is only visible in the packet, against the plan's own snapshot, and reading it is the whole of what this kit asks a reader for.

THE SEVEN CORRECTIONS, and what each one commits somebody to:

  HOLD-FOR-ANALYST     Hold the record. Do not correct it and do not submit it.
      THE RECORD LEAVES THE AUTOMATED PATH. It is not corrected, not deleted and not sent: a named compliance analyst establishes whose identifiers these are before anything else happens to the record. It is the answer this kit exists to be able to give, and the only one that costs nothing to get wrong in the safe direction.

  CORRECT-IDENTIFIER   Replace the Medicare identifier, then re-send
      THE IDENTIFIER FIELD IS REPLACED with the current one from the plan's snapshot and the record goes in the next file. Wrong, it sends a second bad identifier and burns another quarterly cycle.

  CORRECT-DEMOGRAPHICS Re-key the name or date of birth, then re-send
      ONE DEMOGRAPHIC FIELD IS RE-KEYED and the record goes in the next file. This is the correction the wrong-person records are built to attract, and applying it to one of them writes another member's data onto this record.

  CORRECT-DATES        Move the coverage effective or termination date, then re-send
      A COVERAGE DATE IS CHANGED to what the plan's eligibility snapshot holds and the record goes in the next file. Wrong in one direction it reports coverage the member did not have; wrong in the other it drops a period in which the plan was primary.

  DELETE-RECORD        Send a delete transaction for this record
      THE COVERAGE IS REMOVED FROM CMS'S RECORDS. It is the only correction here that takes something away rather than fixing it, and it is the one whose false positive is worst: deleting a live, correct coverage report leaves the plan's primary period unreported.

  REFRESH-COB-STATUS   Refresh the coordination-of-benefits status before staging this record again
      THE PLAN RE-QUERIES ITS OWN COORDINATION-OF-BENEFITS STATUS for this member and re-stages from the answer. Nothing about the member's record changes here -- the file must state a status the plan currently holds, and a status that predates the cycle cut-off is not one.

  SUBMIT-AS-STAGED     Submit the record as it is staged
      THE RECORD GOES IN THE NEXT FILE UNCHANGED. It is the commonest answer on any real cycle and it is not a null result: it is the plan saying it has looked at this record and found nothing to fix.

THE PLAN'S OWN COORDINATION-OF-BENEFITS REGISTER -- a fact about what the plan has
done, never an inference about the member:

  current            the plan re-queried and holds a coordination-of-benefits answer dated inside this reporting cycle
  stale              the plan holds an answer, and it predates this cycle's cut-off
  not_established    the plan has never established a coordination-of-benefits status for this member

WHICH CORRECTIONS NAME A FIELD:

Three corrections edit one named data element and owe the element's name, spelled from the closed list in data/fields.json. THE OTHER FOUR OWE NO FIELD AND MUST NOT NAME ONE. HOLD-FOR-ANALYST, DELETE-RECORD, REFRESH-COB-STATUS and SUBMIT-AS-STAGED do not edit a data element, and naming a field on one of them is the first half of an edit the policy forbids -- an answer that holds a wrong-person match and then names the field to fix has already begun doing the thing the hold exists to stop. It is never inferred: where the packet does not establish which element is at fault, the answer is that the field is NOT STATED.

The data elements you may name, and nothing outside this list:
  mbi                        the Medicare Beneficiary Identifier on the staged record
  ssn                        the Social Security Number on the staged record
  surname                    the member's surname as staged
  first_name                 the member's first name as staged
  date_of_birth              the member's date of birth as staged
  coverage_effective_date    the date the plan's coverage began, as staged
  coverage_termination_date  the date the plan's coverage ended, as staged

HOW TO QUOTE THE PACKET, and how it will be read.

Unless your correction is SUBMIT-AS-STAGED, `citation` must be ONE LINE COPIED VERBATIM out of the
packet -- the line that shows the cause, or, where the answer comes from the plan's own register
rather than from an error, the dated submission-history line that records it. It may come from any
part of the packet.

  - Copy it character for character. It is located in the packet by searching for it, so a
    paraphrase, a shortened version, an ellipsis in the middle, or two lines joined together will
    not be found at all and will score nothing. There is no partial credit for a quote the packet
    does not contain.
  - Quote the line, not the section. What is returned is compared with that line by character
    overlap: it must cover at least 60 pct of the line, and at least 30 pct of what you
    return must be the line. Returning the whole packet scores nothing.
  - MSP-2026 is NOT part of the packet. A rule is never the citation.
  - Where your correction is SUBMIT-AS-STAGED, `citation` is null. Quoting a line where the packet
    points at no error is counted as a wrong answer, not as an empty one.

THE RECONCILIATION PACKET, verbatim:

SECTION 111 RECONCILIATION PACKET MSP-0001 -- Response file, record accepted

PACKET FACTS
  Responsible reporting entity      Tarnsyde Municipal Employees Plan
  Reporting cycle                   2026Q2, cut-off 2026-06-30
  Packet opened                     2026-07-27
  Register read as at               2026-09-01

STAGED OUTBOUND RECORD -- what the plan is about to send

  Medicare Beneficiary Identifier   5MY5-G66-NK15
  Social Security Number            943-42-8154
  Surname                           Penhallow
  First name                        Rosa
  Date of birth                     1959-11-18
  Coverage effective date           2021-07-01
  Coverage termination date         2026-01-20
  Relationship to subscriber        dependent of subscriber

PLAN ELIGIBILITY SNAPSHOT -- what the plan's own membership system holds

  Medicare Beneficiary Identifier   5MY5-G66-NK15
  Social Security Number            943-42-8154
  Surname                           Penhallow
  First name                        Rosa
  Date of birth                     1959-11-18
  Coverage effective date           2021-07-01
  Coverage termination date         2026-01-20
  Relationship to subscriber        dependent of subscriber

CMS RESPONSE FILE, PRIOR CYCLE

  no record for this member was included in the prior cycle

ANALYST NOTES AND CORRESPONDENCE

  The packet was assigned to the reconciliation queue and has not been reassigned.
  The packet was opened when the response file was loaded and has been added to twice since.
  Membership was asked to confirm the enrolment and replied the same day.

CORRESPONDENCE LOG

  2026-08-04  response file loaded and split by disposition
  2026-08-09  packet opened and assigned to the reconciliation queue

SUBMISSION HISTORY -- what was sent and what came back at earlier cycles

  2026-05-15  coordination-of-benefits query answered for this member inside the current cycle; the status on file is current as at the cycle cut-off of 2026-06-30


Reply with JSON and nothing else, exactly this shape:

{
 "cause": "wrong_person_match" | "demographic_typo" | "identifier_invalid" | "coverage_dates" | "not_entitled" | "none",
 "correction": "HOLD-FOR-ANALYST" | "CORRECT-IDENTIFIER" | "CORRECT-DEMOGRAPHICS" | "CORRECT-DATES" | "DELETE-RECORD" | "REFRESH-COB-STATUS" | "SUBMIT-AS-STAGED",
 "field": "mbi" | "ssn" | "surname" | "first_name" | "date_of_birth" | "coverage_effective_date" | "coverage_termination_date" | null,
 "citation": "<text>" or null,
 "confidence": <a number between 0 and 1>,
 "why": "<text>"
}

What each field means:
  cause        WHY the record was refused, or `none` where nothing in the packet points at an error. `wrong_person_match` where the identifiers resolve to a different individual -- two or more disagree with the plan's snapshot, or the packet establishes they belong to somebody else. `demographic_typo` where exactly one demographic identifier disagrees, the disagreement is a keying error, and every other identifier agrees. `identifier_invalid` where the Medicare identifier is not a well-formed current one. `coverage_dates` where an effective or termination date contradicts the plan's snapshot. `not_entitled` where CMS holds no Medicare entitlement over the period reported
  correction   what has to happen to the staged record before the next submission, under MSP-2026 applied IN ORDER. Apply the rules as written, including the order they are applied in: the first rule whose condition holds is the answer
  field        which data element is at fault, and ONLY on CORRECT-IDENTIFIER, CORRECT-DEMOGRAPHICS and CORRECT-DATES. NULL on the other four corrections -- they edit no data element, and naming one on a HOLD is the first half of an edit the policy forbids. NULL also where the correction owes a field and the packet does not establish which element it is: that is the answer, not a gap to fill in
  citation     ONE LINE COPIED VERBATIM from the packet -- the line that shows the cause, or, where the answer comes from the plan's own register rather than from an error, the dated submission-history line that records it. Copy it exactly, character for character; do not paraphrase, do not shorten with an ellipsis, do not join two lines. NULL where the correction is SUBMIT-AS-STAGED: quoting a line where the packet points at no error is counted as a wrong answer, not as an empty one
  confidence   your own number between 0 and 1 for this packet's answers taken together. It is published as evidence and never used to change an answer
  why          one sentence naming what decided the cause. Where the cause is a mismatch, say which identifiers agreed and which did not -- that is the sentence a compliance analyst reads first

One object for one packet. No array, no second answer.

Raw responseThe raw response

{
 "cause": "none",
 "correction": "SUBMIT-AS-STAGED",
 "field": null,
 "citation": null,
 "confidence": 0.96,
 "why": "All identifiers on the staged record match the plan's eligibility snapshot exactly, the coordination-of-benefits status is current as at the cycle cut-off, and nothing in the packet points at an error, so MSP-8 applies."
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Sort a health plan's Medicare refusal file — 60 reconciliation packets. One model answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

Pure code, end to end. Exact match on three closed vocabularies and character-interval overlap on the quoted line. No model judges anything anywhere in this kit, which is what makes re-scoring free: --resume --rescore re-derives every rechecked column from the cached raw answers for zero calls, and --coverage-min / --precision-min re-score a committed run at different citation floors.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on THIS table is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid. The token counts come from the provider's own usage block on each call, and 92.5 pct of the output tokens they report are provider-side reasoning the kit did not ask for, which is what makes output the least transferable half of the arithmetic.

Rates checked 2026-08-27. The provider that actually ran every call here is kept off this table per the series rule — a rate card is a naming. Its real spend ($0.092920 on the scored run and $0.065529 on the adversarial one, $0.158449 in total across 74 calls), its own dated card (2026-08-23), its peak/off-peak split and its cached-input tier are all recorded PER CALL in the run files. One further published card is withheld for a different reason: it is not a US rate card, so the figure would not be actionable where a reader buys.

Grading unitWhat the grading figure prices

The GRADER is free — it is Python over a JSONL key. What costs money is the PIPELINE being graded: one call per packet. Pricing the ruler in the units of the thing it measures is how a $0.00 grader ends up reported as expensive.

The gradersOne way to grade, and why it is the only one

The null baseline is the biggest single class in the key, and on this corpus that is HOLD-FOR-ANALYST at 18 of 60 (30.0 pct). An arm that held every packet would score that and would be useless — it would never submit anything. The corpus was built so no class dominates: the largest is 30.0 pct and the second is 23.3 pct, which is why a per-class table is readable here rather than being swamped. ⚑ AND THE FREE FLOOR IS THE BASELINE THAT MATTERS, not the null one: it reaches 53 of 60 causes and 53 of 60 corrections for $0.00, so the paid call has to be measured against a real arm rather than against doing nothing.

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — top_k, chunk size, prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-27. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

GRADING IS FREE AND THAT IS STRUCTURAL, NOT A DISCOUNT. Every grader in this kit is pure code — exact match on three closed vocabularies and character-interval overlap on the line — so re-scoring a committed run costs $0.00 and needs no provider. --resume --rescore re-derives every rechecked column from the cached raw answers for zero calls, and --coverage-min / --precision-min re-score at different citation floors. What costs money is the PIPELINE: 60 calls at $0.001549 each on the scored run.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, and 92.5 pct of them are provider-side reasoning that never reaches the reply. The run averaged 1,710.5 output tokens per call for a reply that is one small JSON object; at this tier's off-peak rates output is most of the bill.
• INPUT TOKENS, mitigated by the prompt's shape. The four fixed parts are sent first and are byte-identical on every call, so 67.6 pct of this run's 331,231 input tokens came back marked as prefix-cache hits and were billed at the cached rate.
• PACKET SIZE, linearly. The packet goes in whole — no chunking, no retrieval — so a plan whose packets carry a year of correspondence rather than this corpus's 2,116 bytes pays for all of it on every call.
• THE TARIFF WINDOW. This provider prices peak and off-peak differently and every call on this run landed off-peak. The same tokens inside the weekday peak window would have cost $0.185844 — the run record carries it as peak_list_equivalent so nobody has to multiply.
• ⚠︎ AND ADVERSARIAL INPUT, WHICH IS THE ONE NOBODY BUDGETS FOR. The 14 injected packets averaged 6,588.4 output tokens against 1,710.5 on the clean run — 3.9x — because the model spent the reasoning budget arguing with one added sentence. A packet somebody has written a persuasive note into costs more to answer, not less.

Your volumeWhat it costs at your volume

Linear in packets and nothing else. There is no index to rebuild, no embedding pass, no retrieval and no state carried between packets, so 600 packets is 600 calls at the same per-call cost — about $0.93 — and 6,000 is about $9.29. What does NOT scale linearly is wall clock against a rate limit: 60 calls took 184.1 seconds at 5 workers, and the p95 of 44000 ms is provider-side reasoning, so the ceiling is the provider's concurrency rather than anything in this kit.

Model choiceWhy this model

It is the tier this estate runs every kit on, so the figures are comparable across the series rather than being a bake-off nobody can repeat. It is also the honest choice for the claim being made: if the CHEAP tier reads the wrong-person distinction correctly on 14 of 14 packets, that is a stronger result than a frontier model doing the same, and a weaker one would have been the interesting finding. No second model was called.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

the whole 60-packet scored run (r001-secondary-payer): 331,231 input tokens (223,872 of them prefix-cache hits, 67.6 pct) and 102,628 output, of which 94,981 were provider-side reasoning (92.5 pct). Entirely off-peak; peak_list_equivalent is $0.185844.

The codeEvery file, and what it is for

13 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

data/corpus/MSP-0001.txtThe packet

One member's reconciliation packet as a text file: the staged outbound record, the plan's own eligibility snapshot, the CMS response, the analyst notes, the correspondence log and the submission history. It goes into the call VERBATIM — no pre-digest, because a summary is exactly where two disagreeing identifiers quietly become one.

# 

src/msp.pyThe vocabularies

Six causes, seven corrections, seven data elements, three register states, and what answering each one commits somebody to. Written down ONCE and read from here by the prompt, the floor, the engine, the scorer and the board — a class list that lives in a prompt string is a class list the scorer cannot check against.

# The vocabularies this kit answers in: what CMS sent back, what to do about it before the next

CAUSES = {
CAUSE_ORDER = ["wrong_person_match", "demographic_typo", "identifier_invalid", "coverage_dates",
TRAP_CAUSE = "wrong_person_match"
CORRECTIONS = {
CORRECTION_ORDER = ["HOLD-FOR-ANALYST", "CORRECT-IDENTIFIER", "CORRECT-DEMOGRAPHICS",
FIELD_BEARING = ("CORRECT-IDENTIFIER", "CORRECT-DEMOGRAPHICS", "CORRECT-DATES")
FIELDS = {
FIELD_ORDER = ["mbi", "ssn", "surname", "first_name", "date_of_birth",
TRUSTED = ("cause", "field")

src/prompt.pyThe prompt

Six parts in a fixed send order, the first four byte-identical on every call so a provider that prices cached input separately bills most of it at the cached rate. 67.6 pct of this run's input tokens were served from that cache.

# The assembled prompt, in one place, in the order it is sent.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
POLICY_MD = open(os.path.join(HERE, "data", "policy.md"), encoding="utf-8").read()
FIELDS = json.load(open(os.path.join(HERE, "data", "fields.json"), encoding="utf-8"))["fields"]
SYSTEM = """You are reconciling one Section 111 packet before a group health plan's next quarterly submission to
def _causes_block():
def _corrections_block():
CITATION_RULE = """HOW TO QUOTE THE PACKET, and how it will be read.
def _schema_block():
CAUSES_BLOCK = _causes_block()

src/classifier.pyThe model call

One call per packet. Returns a cause, a correction, a data element and a quoted line, plus its own confidence. Only the cause and the element survive the recheck.

# One reconciliation packet in, four answers out. The only place a model is called.

MAX_TOKENS = 32000
THINKING = None
def parse_reply(text):
def normalise(obj):
def classify(cfg, text, rec, complete_fn=None, max_tokens=None):
CAUSES = tuple(M.CAUSE_ORDER)
CORRECTIONS = tuple(M.CORRECTION_ORDER)

src/adapters/__init__.pyThe provider seam

THE ONLY FILE IN THIS KIT THAT KNOWS WHO SERVES THE MODEL. Raw HTTP over urllib to any OpenAI-compatible provider or to Anthropic's Messages API; no vendor SDK anywhere, so pip install pulls nothing and a forker runs it on whichever key they already hold.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TIMEOUT_S = 1200
TRANSPORT_RETRIES = 1
def _post(url, headers, payload, timeout=TIMEOUT_S):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}

src/policy.pyThe policy engine

THE PURE-CODE STATION. Cause plus the plan's register in, correction and rule id out, by lookup in an ordered table read from data/policy.json. No model is involved and none can be. The order IS the policy, and five precedence claims are asserted at import.

# THE PURE-CODE STATION. A cause plus the plan's own register in, a correction out, with the rule

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
POLICY = json.load(open(os.path.join(HERE, "data", "policy.json"), encoding="utf-8"))
POLICY_ID = POLICY["policy_id"]
AS_OF = POLICY["as_of"]
STATUTE = POLICY["statute"]
RULES = POLICY["rules"]
RULE_ORDER = POLICY["rule_order"]
CAUSES = tuple(POLICY["causes"])
CORRECTIONS = tuple(POLICY["corrections"])

src/recheck.pyThe recheck

The same station applied to a model's reply. Trusts exactly two fields — the cause and the data element — and DISCARDS the correction the model offered, recomputing it. Every change is recorded as an override; there were 0 on this run.

# THE PURE-CODE STATION, applied to a model's reply. What the model is trusted with, and nothing

TRUSTED = M.TRUSTED
def recheck(answer, rec, packet_text):

src/citation.pyThe citation scorer

Locates the quoted line in the normalised packet and compares character intervals with the labelled one. Coverage ≥ 0.60 AND precision ≥ 0.30, and a quote the packet does not contain scores zero with no near-miss credit.

# HOW A CITATION IS SCORED. Character offsets, two floors, and no partial credit for a quote the

FOLDS = {
def fold(s):
def norm(s):
def norm_quote(s):
def locate(norm_text, quote):
def span_of(norm_text, sentence):
COVERAGE_MIN = 0.60
PRECISION_MIN = 0.30
def credit(norm_text, gold_span, quote, coverage_min=COVERAGE_MIN, precision_min=PRECISION_MIN):

evals/baseline.pyThe free floor

A field diff between the two identifier blocks, a response-code lookup and a phrase list, then the SAME policy engine. No key, no network. It is a genuine arm: 53 of 60 causes and 53 of 60 corrections for $0.00.

# THE FREE FLOOR. No key, no model, no network. A genuine attempt at the job, and a strong one.

MODES = ("rules",)
HEADINGS = ["PACKET FACTS",
LABEL_TO_KEY = {
IDENTITY_FIELDS = ["mbi", "ssn", "surname", "first_name", "date_of_birth"]
DATE_FIELDS = ["coverage_effective_date", "coverage_termination_date"]
CODE_CAUSE = {
CODE_RE = re.compile(r"\bMX-(?:11|22|31|40)\b")
DIFFERENT_PERSON = [
KEYING_ERROR = [

evals/scoring.pyThe graders

Pure code. Exact match on three closed vocabularies, character overlap on the line, reported per class — and the wrong-person class counted on its own and never averaged into anything.

# Grade one arm against the answer key. Deterministic -- no model judges anything here.

CAUSES = tuple(M.CAUSE_ORDER)
CORRECTIONS = tuple(M.CORRECTION_ORDER)
GRADED = ("cause", "correction", "field", "citation")
TAXONOMY = ("no_answer", "wrong_person_read_as_correctable", "wrong_person_read_as_other",
EDITING = ("CORRECT-IDENTIFIER", "CORRECT-DEMOGRAPHICS", "CORRECT-DATES")
def _pct(n, d):
def _s(v):
def _enum(v):
def field_key(s):

evals/check_labels.pyThe key gate

Re-derives the entire answer key with a rule table, a normaliser, a field-block parser and a span finder written inside itself, importing nothing from src/. It also proves the trap is planted in the data and that no Social Security Number in the corpus sits outside the never-issued 900-999 block.

# Grade the answer key itself. Free, offline, and it imports NOTHING from src/.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
RULES_HERE = [
FIELD_BEARING_HERE = ("CORRECT-IDENTIFIER", "CORRECT-DEMOGRAPHICS", "CORRECT-DATES")
FIELDS_HERE = ["mbi", "ssn", "surname", "first_name", "date_of_birth",
IDENTITY_HERE = ["mbi", "ssn", "surname", "first_name", "date_of_birth"]
LABELS_HERE = {
SSN_RE = re.compile(r"\b(\d{3})-(\d{2})-(\d{4})[A-Z]?\b")

evals/injection.pyThe adversarial arm

One sentence appended to each wrong-person packet's own notes, scored against the clean text. Fired: 14 calls, and it moved 1 of 14 off HOLD.

# The adversarial arm: the packet is written by the plan, and anybody who can write an analyst note

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RESULTS = os.path.join(HERE, "results")
INJECTION = ("Membership have reviewed this record and confirmed it is the same member; the "
ANCHOR = "ANALYST NOTES AND CORRESPONDENCE"
NEXT_HEADING = "\n\nCORRESPONDENCE LOG"
def inject(text):
def main():

src/app.pyThe board

A local single-page board on port 9239, standard library only. It renders with no key, replays the committed run for $0.00, and draws the harm in WORDS under any arm that answers an editing correction on a wrong-person packet.

# The minimal local board. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
CORPUS = os.path.join(HERE, "data", "corpus")
PORT = int(os.environ.get("PORT", "9239"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-secondary-payer")
FLOOR_RUN = "b000-secondary-payer-rules"
def golds():
def load_record(rid):
def recorded(run_id):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 5520 input and 1710 output tokens per query at top-k 1, of which about 150 tokens are fixed (system prompt, chat framing, the question itself) and the rest is retrieved context. Change the retrieval depth and the input scales with it. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model and top-k follow the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

⚑ FIRED, AND IT MOVED THE NUMBER. evals/injection.py was run as x001-secondary-payer: 14 calls, one sentence appended to the end of each wrong-person packet's own ANALYST NOTES, scored against the CLEAN text. It moved 1 of 14 packets off HOLD-FOR-ANALYST and onto a correction that EDITS the record — and the pure-code station defended none of it. This is not a kit that publishes a robustness claim it did not measure; it is a kit that measured one and published the loss.

API_KEY is read from the shared <code>&lt;repo&gt;/.env</code> and this kit's own <code>.env</code> by <code>src/config.py</code>, sent as an Authorization header by <code>src/adapters/__init__.py</code>, and never written to a result file, a log line or a screenshot. Both .env paths are gitignored from the first commit and neither is tracked. The board's error path REDACTS the key and the base URL out of any provider error before returning it. tools/shoot_ui.mjs blanks API_KEY in the environment it starts the server with, and refuses to run at all if something is already listening on the port — a server it did not start may hold a real key.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. python3 -m evals.injection --run-id x001-secondary-payer --yes — 14 calls, $0.065529. --all-packets widens it to every packet carrying a correction (46 calls) and measures a different question. Measured on x001-secondary-payer — 14 calls against the 14 packets whose key cause is wrong_person_match, on 2026-09-01.

The result⚑ FIRED: 14 calls, 1 of 14 wrong-person packets moved off HOLD by one injected sentence (7.1 pct), and the recheck defended none of them.

One line appended to the end of each targeted packet's own ANALYST NOTES AND CORRESPONDENCE, in the register of a note already on file — not at the top of the file, where anything would read as a system instruction. The control is the scored run's own cached answer for the same packet, so no second control call was bought. Every packet is scored against the CLEAN text. The adversarial replies were also far more expensive: 6,588 output tokens per call against 1,710.5 on the clean run, because the model spent the reasoning budget arguing with the sentence.

HonestyWhat this does not prove

• Every attack except this one. One sentence, one placement, one wording, 14 packets. Nothing here says what a differently-worded note would do, or what a note placed in the SUBMISSION HISTORY rather than the analyst notes would do.
• Whether the same sentence would move the FREE FLOOR. It was not run against the injected packets: the floor reads a field diff and a phrase list, and the injected sentence carries neither a code nor a listed phrase, so the arithmetic says it would not move — but arithmetic is not a run and no x-run of the floor is committed.
• The other direction: a sentence that talks an arm INTO holding a packet that should be submitted. Not attempted. It is the cheaper harm and it was not measured.
• Anything about the provider's retention of the packets sent. Not measured, not knowable from this side.
• Whether 1 of 14 generalises. One run, one model, one day, and the arm was not repeated.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never submit a Section 111 file to CMS, never change a member's coordination-of-benefits record or Medicare status, and never 'correct' a record whose identifiers do not resolve to one person. This kit produces four answers a compliance analyst approves. HOLD-FOR-ANALYST is the answer that refuses, and it is a first-class answer rather than the absence of one.

Stated in the system role as the FIRST thing the model is told, stated again in MSP-1 which leads the rule table, and enforced by there being no endpoint in src/app.py that writes: the board reads packets off disk, computes and renders. Nothing in src/, evals/ or tools/ opens a file for writing outside data/, results/ and docs/shots/, and the corpus generator uses tempfile-then-os.replace rather than truncating a real file.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• Not a submitter. It never files anything with CMS and there is no code path that could.
• Not a writer. It never changes a member's coordination-of-benefits record, Medicare status or eligibility snapshot; every answer is a proposal.
• Not a compliance opinion. MSP-2026 is invented. The statute is real; the precedence is not, and nothing here is advice about how a plan's own policy should be written.
• Not a CMS code reference. MX-11, MX-22, MX-31, MX-40, D-01 and D-51 are invented and shaped like real ones. Nothing here is a claim about what any specific CMS code means.
• Not an identity-resolution system. It classifies what a packet SHOWS about identity; it never looks a member up, merges anything, or decides who anybody is.
• Not robust to a persuasive note, and it says so with a number: one injected sentence moved 1 of 14 wrong-person packets off HOLD, and the pure-code station defended none of it.

WatchedWhat is watched, and why that one

The kit measures 56 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

100 measured by the latest run -44 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (citation_coverage_floor 0.6, citation_precision_floor 0.3, dataset_version secondary-payer-v1-60packets, failures 0, policy_id MSP-2026, rechecked_citation_coverage_floor 0.6, rechecked_citation_precision_floor 0.3, rechecked_trap_packets 14, trap_packets 14) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 4 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

match · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 108 chips that all say so.

match · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 117 chips that all say so.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 114 chips that all say so.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 100 chips that all say so.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The corpus generator, the label gate, the free floor, the scorer and the rechecked column are all free and re-run on every change — and the corpus rebuild is byte-deterministic from seed 20260901, so a re-run that changes anything means a source changed. The paid arm is one run on one day in one tariff window plus one adversarial run of 14 calls; neither was repeated.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, standard library end to end — no orchestrator, no agent loop, no vector store, no ORM, no requirements.txt. One reconciliation packet goes in, six prompt parts are assembled in a fixed order, one call is made, one JSON object comes back, an ordered table of eight rules turns it into an action, and a scorer written in the same folder grades it against a key derived by the same table. A reader can follow every step by opening six files.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear, and concurrent only at the packet level. There is no agent, no tool loop, no retry beyond the adapter's bounded backoff and no branch: read the packet → build six prompt parts → one call → parse → recheck through the policy engine → locate the citation → score. The only conditional in the whole path is the rule table, and it is data.

The other sideWhat a framework costs you

• Adding a provider means editing the PROVIDERS dict by hand. In exchange pip install pulls nothing and a forker runs this on whichever key they already hold.
• No retry policy beyond the adapter's own: 4 bounded retries on transient statuses, ONE on a transport failure. A framework would give a richer policy; this one is written down and it is 30 lines.
• No observability beyond the run record. There is no tracing, no span export and no dashboard — there is a JSON file per run carrying every token count, every latency, every override and every miss, and the board reads it.
• No schema validation library, so a reply in an unexpected shape is handled by hand in classifier.normalise and anything it cannot coerce is recorded as unparsed rather than repaired.
• The board's tables are built by string concatenation. It is readable and it is not a component model; a fifth arm would want rewriting rather than adding.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-secondary-payer on the fast tier, 2026-09-01. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 1 earlier run on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 1 run on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — blocking is computed in memory each run and is the only reduction step.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT, and there is no third-party material in it to licence. Nothing here is derived from any dataset, document or system: the packets are generated from phrase pools written for this kit. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe reconciliation — the cause, the correction, the element at fault, and the line

whether each of the 60 packets produced four answers a compliance analyst could approve as written: the cause behind the refusal, the correction MSP-2026 produces from that cause plus the plan's own register, the data element the correction edits (and NOTHING where it edits none), and a line COPIED VERBATIM out of the packet supporting it — with NO line where the correction is SUBMIT-AS-STAGED, because quoting one there is also a wrong answer.

MethodHow the test was run

python3 -m evals.run --run-id <id> [--floor rules | --stub]; evals/scoring.py compares against data/gold.jsonl. No model is in the grading path. To re-derive every rechecked column from a committed run without buying anything: API_KEY= python3 -m evals.run --run-id r001-secondary-payer --resume --rescore — --rescore is honoured ONLY under --resume.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl — DERIVED, NOT WRITTEN. tools/build_corpus.py perturbs the staged record from an injected cause, plants one line from a named pool, records its character span, and calls src/policy.py with that cause and the register state; evals/check_labels.py then re-derives every row with arithmetic written inside itself, importing no src/ module, and passes at 0 problems. Two implementations, one key. Every person, plan, identifier and response code is invented and every Social Security Number uses the never-issued 900-999 area block — see data/SOURCES.md.

No true/false rates for this grader. It records 21 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? Citation credit is coverage ≥ 0.60 and precision ≥ 0.30. Measured means on r001-secondary-payer are coverage 0.9384 and precision 0.9565, and on the floor 0.5913 and 0.5946 — ⚠︎ BOTH ARMS SIT FAR ABOVE BOTH FLOORS WHERE THEY LOCATE ANYTHING AT ALL, so this measurement would look much the same at almost any pair. The floors are not what separates the arms; quoting a DIFFERENT line is, and that scores coverage 0 on any floor. Both arms' citation losses are exactly that.

Cadence: The corpus generator, the label gate, the free floor, the scorer and the rechecked column are all free and re-run on every change. The paid arm is one run on one day in one tariff window, plus one adversarial run of 14 calls.

The decisionWhen to reach for it