The business caseThe problem this solves

CMS returns a transaction reply file against the enrollment transactions a plan submitted. Each line carries a reply code, the beneficiary segment CMS holds and the transaction as it went, and somebody has to decide what actually went wrong. THE LOOKUP IS NOT THE JOB: a reply code names its own root cause only some of the time, and where it names two an analyst has to read the transaction against the segment before anyone can file anything. The failure that costs money is never a wrong lookup. It is a reject clustered under the wrong root cause -- which sends the correction down a queue where nobody re-files anything, so the enrolment simply does not exist -- and a corrective transaction drafted with an effective date that re-rejects, which looks filed, clears the worklist, and comes back next month saying the same thing. Neither reports anything downstream. An enrollment analyst reading each reply line against the segment CMS holds -- comparing the package submitted against the packages open at that segment, deciding whether a name that differs by a hyphen is a demographic difference or nothing at all, working out whether the effective date requested clears BOTH entitlement starts and the retro window this file opens, noticing that one reply code is exempt from that window, and then reading the submitting desk's own note on the lines where two explanations are both true and nothing printed decides between them.

The inputThe actual reply-file packs

The corpus is 60 reply-file packs, 0.39 MB (txt 60). ⚑ A TRANSACTION REPLY FILE CANNOT BE PUBLISHED, BY ANYBODY, EVER. It is a per-beneficiary record of enrolment activity between a plan and CMS, and the fields being reconciled -- the identifier, the entitlement dates, the demographics as submitted -- ARE the sensitive part. There is no public corpus, no redaction that would make one publishable, and there never will be one to fetch. And the prose is not what is measured: what is measured is a decision and the line behind it, and both need facts that are KNOWN -- which root cause, which queue, which effective date to the day, which lines cannot be settled at all, and exactly which line establishes it, to the character. So the facts are planted, the key is DERIVED from them by src/policy.py, and the packs are written around them.

TRANSACTION REPLY FILE PACK RR-0001 -- daily reply file, afternoon batch

PACK FACTS
  Submitting contract                H9858
  Reply file received                2026-03-05
  CMS processing month               2026-02
  Retro window                       3 months -- the earliest effective date a corrective
                                     transaction may carry is 2025-11-01, unless the reply
                                     code is marked retro-eligible below
  Reconciliation procedure           RR-2026, read as at 2026-08-31

REPLY CODE TABLE IN FORCE -- RC-2026, read as at 2026-08-31

  R-101   accepted   Enrollment accepted as submitted
                     NOT_A_REJECT
  R-104   accepted   Disenrollment accepted as submitted
                     NOT_A_REJECT
  R-107   accepted   Plan benefit package change accepted as submitted
                     NOT_A_REJECT
  R-212   settled    Identifier submitted is not on file
                     IDENTIFIER_INVALID
  R-224   settled    A transaction covering this period is already on file
                     DUPLICATE_TRANSACTION
  R-241   settled    Beneficiary is enrolled with another organisation for the period requested
                     OTHER_PLAN_ENROLLMENT
  R-256   settled    Effective date requested is outside the window this transaction accepts
                     EFFECTIVE_DATE_INVALID; RETRO-ELIGIBLE
  R-231   ambiguous  Beneficiary record does not support the transaction as submitted
                     consistent with DEMOGRAPHIC_MISMATCH or PLAN_CODE_MISMATCH -- the transaction as submitted decides
  R-238   ambiguous  Coverage requested cannot start on the date submitted
                     consistent with EFFECTIVE_DATE_INVALID or ENTITLEMENT_GAP -- the transaction as submitted decides

The outcomeWhat a good result looks like

A drafted worklist: one entry per reply line, in the pack's order, each carrying the root-cause cluster, the queue it goes to, the corrective effective date where a correction is drafted, and the line of the pack that establishes it -- plus the three free floors' answers beside every one.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• your reply codes each name one root cause and your desk keys them by hand — the reply-code lookup floor (b000), and no model at all
  it takes 110 of 240 lines here for $0.00 and every one of them is a table read. Paying a model for a lookup is paying for a lookup.
• your reconciliation tool already prints field-by-field comparisons — the printed-comparison floor (b001)
  204 of 240 lines, $0.00, and it takes the plan-code trap 40 of 40 because RR-9 is a written rule a floor author would read.
• the deciding evidence on your hard lines is FREE TEXT somebody typed — the model call -- but measure the channel first
  36 of these 240 lines are settled only by prose, and the fast tier takes all 36. That is the only band on this corpus where a call is doing something a lookup cannot.
• you want the drafted effective date to be right — pure code -- src/policy.py::corrective_date()
  RR-5..RR-8 are max() over four printed dates. Every arm here scored 0 of 51 on date_would_rereject, and src/recheck.py re-derives the date from the model's own reading so a fluent wrong date cannot survive.

At a glanceHow the whole thing runs

Run once, for real, on 2026-09-01. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-09-01 — r001-reply-reconcile. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A clean checkout with no key configured renders the whole board on 127.0.0.1:9236, rebuilds the corpus to the byte, re-derives the answer key independently, and scores all three free floors offline -- 60 packs, 240 judgements, $0.00. The committed scored run replays from its result file with every quote highlighted at its offsets. Only the "Check with the model" control needs a key, and with none it is disabled and says so.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per pack, five concurrent workers against a shared key that sibling kits were using at the same time; 604.9 s of wall clock for 60 packs. Read it as a bound, not as a clean single-tenant latency.

Current processWhat it replaces

An enrollment analyst reading each reply line against the segment CMS holds -- comparing the package submitted against the packages open at that segment, deciding whether a name that differs by a hyphen is a demographic difference or nothing at all, working out whether the effective date requested clears BOTH entitlement starts and the retro window this file opens, noticing that one reply code is exempt from that window, and then reading the submitting desk's own note on the lines where two explanations are both true and nothing printed decides between them.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one reply lines (4 per pack) is

No train/test split, because nothing is trained or tuned, and no segmentation, because one pack goes whole into one call (the sizes here are the packs' own bytes; mean 6,800.4). THE UNIT EVERY RATE STANDS ON IS ONE REPLY LINE — and there are 240 of them across 60 packs; every arm is scored over all 240 whether or not it answered, and a reply that returned three rows has failed to answer the fourth rather than answered NOT_A_REJECT on it. The split that matters here is the EVIDENCE CHANNEL, not a train/test one: 110 lines the reply code table settles, 94 one printed comparison settles, and 36 that only the submitting desk's own note can. 13 named cases: accepted_line 60, ambiguous_plan_code 20, ambiguous_plan_code_note 20, ambiguous_entitlement 18, settled_date_window 16, ambiguous_demographic_note 16, ambiguous_date_window 14, ambiguous_demographic 14, ambiguous_pbp_or_identifier 14, unresolved 14, settled_identifier 12, settled_duplicate 12, settled_other_plan 10. ⚠︎ EVERY ONE OF THE 60 PACKS CARRIES AT LEAST ONE LINE BUILT TO BE GOT WRONG, so clean_packs is 0, clean_all_correct_pct is null and hard_all_correct_pct is identical to pack_all_correct_pct. That denominator is degenerate and it is published that way rather than quietly removed.

SetupWhat the setup figure measured

THERE IS NO INDEX IN THE RETRIEVAL SENSE and nothing to retrieve from: one pack goes whole into one prompt at a mean of 6,981.4 input tokens. What this kit reduces with instead is the REPLY CODE'S CLASS, read from data/reply_codes.json in memory on every run — a code that names its own root cause makes the reading irrelevant, and that is 110 of the 240 lines. data/packs.json is the reply file as structured data, read by src/policy.py for the processing month, the retro window and each line's printed dates, and never sent as a retrieval step. The whole free half costs nothing and takes no measurable time: results/eval-b000-reply-reconcile-codetable.json carries wall_seconds 0.4 for answering and scoring all 240 judgements with no key and no network. ⚠︎ THE BUILD FIGURES ABOVE ARE ZERO BECAUSE THERE IS NOTHING TO BUILD, not because a build was fast and nobody timed it.

LicenceLicence

MIT, as the rest of this repo. Every pack, contract, identifier, name, date and reply code in it is invented, so there is nothing in it anybody else holds rights over.

Bring your ownBring your own reply-file packs

Replace data/packs.json with your own reply-file rows -- one object per pack, four lines each, carrying the reply code, the beneficiary segment and the transaction as submitted -- and rewrite tools/build_corpus.py's render() to print them in your file's own layout. Your reply codes go in data/reply_codes.json with their class and, where ambiguous, their two candidates; your procedure goes in data/policy.json and its prose twin data/policy.md. Nothing under src/ needs to change: the engine reads the rulebook, the scorer reads the key, and evals/check_labels.py will tell you if the two renderings of your procedure disagree before you spend anything.

⚠︎ And what stops being true when you do: ⚠︎ WHAT STOPS BEING TRUE ON YOUR CORPUS IS EVERY NUMBER ON THIS PAGE. The margins here are measured against findings and desk notes drawn from a small template pool, and 15 pct of these lines are settled only by prose because that is what was planted. A real reply file's notes are written by whoever worked the batch, and the channel split on your own desk is the first thing you would have to measure before any of this transfers.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 29,586 characters assembled, 7,940 are policies — 27% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Replayed from the kit's own src/prompt.py::build() against pack RR-0001, which is only trustworthy because the replayed part list and character counts match what the run recorded: six parts, 29,586 characters on this pack, sections system/policy/root_causes/actions/pack/schema. ⚠︎ THE PART SIZES ARE PUBLISHED IN CHARACTERS, NOT TOKENS. The provider reports one input total per call and no per-part split, so a per-part token count would be an apportionment nobody measured. The measured totals are in tokens.

Full promptThe prompt in full

You are working one transaction reply file returned by CMS to a Medicare Advantage plan, and
answering a single operational question about each line on it: what actually went wrong with the
transaction the plan submitted, and what -- if anything -- should be drafted to correct it. Your
output is the worklist an enrollment analyst reviews, so that the analyst reviews four answers
instead of re-reading the pack.

THIS IS A RECONCILIATION AND DRAFTING QUESTION AND NOTHING ELSE.

  - YOU NEVER SUBMIT ANYTHING TO CMS. Where a corrective transaction is warranted you DRAFT one --
    a root cause and an effective date -- for an enrollment analyst to review and submit. Nothing
    in this system reaches CMS and nothing in it may claim to have filed anything.
  - YOU NEVER CHANGE THE PLAN'S OWN ENROLLMENT RECORD. A reply code is evidence about what CMS did
    with a transaction. It is not an instruction to write to the system of record, and no reading
    of one authorises a change to a beneficiary's enrolment.
  - Say nothing about the beneficiary beyond what the reply file prints. The packs carry no
    clinical information and no reason for any enrolment; a reply that offered one would be
    inventing it.

How to read the pack:

- THE REPLY CODE TABLE IN FORCE IS PRINTED ON THE PACK. Use that table and no other. A code's
  CLASS in it is what decides whether your reading matters at all.
- Each REPLY LINE carries the beneficiary segment CMS holds, the transaction as the plan submitted
  it, and a short list of RECONCILIATION FINDINGS -- plain statements of fact about the fields
  above them. A finding never states a conclusion; it states a comparison.
- THE FINDINGS ARE NOT RANKED AND MORE THAN ONE MAY BE TRUE. A line can carry a real error and a
  harmless difference at the same time, and the harmless one is often printed first. RR-9 is what
  tells them apart and it is in the procedure below.
- ⚑ AND SOMETIMES BOTH FINDINGS ARE REAL ERRORS. Where a line carries two comparisons that are each
  genuinely true -- a surname that really is different AND a package that really is not open -- no
  comparison of the printed fields can say which one CMS rejected on. THE DESK NOTE ON THAT LINE IS
  WHAT SETTLES IT, and it settles it by saying what has already been reconciled and accepted, never
  by naming a cause. A note that says the identity was reconciled with CMS and has been accepted
  since leaves the package as what came back; a note that says the package was accepted for other
  members leaves the identity. Read the note before you decide a line whose findings both hold.
- EVERY REPLY LINE CARRIES A DESK NOTE, so the presence of one means nothing. On most lines it
  settles nothing at all.
- THE PACK-LEVEL ANALYST WORKING NOTES AT THE END ARE COMMENTARY, not evidence. They are somebody's
  remarks about the batch as a whole. Nothing in them establishes a root cause for any line and
  nothing in them may be quoted as one. The per-line DESK NOTE is a different thing and it is
  evidence.
- Answer all four reply lines, once each, in the order the pack prints them. Most reply files carry
  at least one accepted line; answering NOT_A_REJECT on it is the normal shape of a reply here.
- Apply RR-2026 as written, INCLUDING THE ORDER ITS RULES ARE APPLIED IN.
- THE CORRECTIVE EFFECTIVE DATE IS ARITHMETIC, NOT JUDGEMENT. Every date it needs is printed on
  the pack. Work it out; do not carry over the date the transaction originally requested unless it
  already clears every floor that applies.
- Give one confidence between 0 and 1 for the pack's four answers taken together.

Reply with JSON and nothing else, in the shape given at the end.

RR-2026, THE RECONCILIATION PROCEDURE, as approved:

# RR-2026 — working a transaction reply file

**This procedure is invented and it is not a regulation.** It is a plan's own operating procedure
for reconciling a transaction reply file against the enrollment transactions that produced it,
written so that its precedence can be checked line by line. **It carries no regulatory citation on
purpose** — reconciling a reply file is an operational job, not a codified provision, and a
citation invented to make it look official would be worse than none at all.

The unit is **one reply line**. A reply file comes back with one line per transaction the plan
submitted; each line carries a reply code, the beneficiary segment CMS holds, and the transaction
as it was submitted. Every line is answered on its own.

Three things are produced for each line: **the root cause**, **what to do about it**, and, where a
corrective transaction is drafted, **the effective date that transaction should carry**.

**Nothing in this procedure files anything.** A corrective transaction is DRAFTED for an enrollment
analyst to review and submit. Nothing here writes to the plan's own enrollment record either: a
reply code is evidence about what CMS did with a transaction, and it is not an instruction to
change the system of record.

---

## The root cause — rules RR-1 to RR-4, applied in this order

**RR-1 — An accepted reply code is not a reject → NOT_A_REJECT**
Where the reply code table classes the code as an acknowledgement, the line is `NOT_A_REJECT`
whatever else the pack says about it. Nothing is drafted, nothing is queued and no date is
produced. **This rule is first and that is the point:** a reply file is not a reject file, and no
reading of a transaction may turn a line CMS accepted into one it did not.

**RR-2 — A reply code the table settles governs the root cause → the table's own cluster**
Where the code table maps the code to exactly one root cause, that root cause is the answer and
the transaction as submitted does not change it. This is the free half of the job: a lookup
settles it, and the reading is not what is being paid for.

**RR-3 — Where the table leaves two candidates, the transaction as submitted decides → the
candidate the transaction shows**
Where the code table names two possible root causes, the root cause is the one the transaction as
submitted, read against the beneficiary segment CMS holds, actually shows — and only where that is
one of the two the code allows. A reading that names a cause the code does not allow has not
settled the line, and RR-4 takes it instead. **This is the rule the whole procedure turns on**, and
it is the only place in RR-2026 where a reading changes an answer.

**RR-4 — Where nothing settles it, the line goes to an analyst → UNRESOLVED_NEEDS_ANALYST**
Where the code leaves two candidates and the pack carries nothing that decides between them, the
line is `UNRESOLVED_NEEDS_ANALYST`. It is a real answer and not a failure to answer: it names the
line, it names what could not be settled, and it declines to file a corrective transaction on a
guess. Guessing here is not free — a corrective transaction filed on the wrong root cause
re-rejects, and the month it spends doing so is a month of coverage nobody has.

### What each root cause commits somebody to

| root cause | what happens next |
|---|---|
| `UNRESOLVED_NEEDS_ANALYST` | a person reads the line; nothing is drafted |
| `NOT_A_REJECT` | nothing at all |
| `PLAN_CODE_MISMATCH` | a corrective enrollment transaction is drafted with a package open at this segment |
| `DEMOGRAPHIC_MISMATCH` | a person settles the demographic with CMS; **re-filing the same transaction re-rejects** |
| `IDENTIFIER_INVALID` | a person resolves the identifier; nothing is filed under the one that was rejected |
| `EFFECTIVE_DATE_INVALID` | a corrective enrollment transaction is drafted with a corrected effective date |
| `ENTITLEMENT_GAP` | a corrective enrollment transaction is drafted, and it cannot begin before entitlement does |
| `DUPLICATE_TRANSACTION` | nothing is filed; CMS already holds one |
| `OTHER_PLAN_ENROLLMENT` | a person works the competing enrolment; re-filing does not resolve it |

### What to do about it — the three actions

Every root cause takes exactly one action, and it is not a choice:

| action | what it means | root causes that take it |
|---|---|---|
| `RESUBMIT` | a corrective enrollment transaction is **drafted**, with an effective date, for an enrollment analyst to review and submit | `PLAN_CODE_MISMATCH`, `EFFECTIVE_DATE_INVALID`, `ENTITLEMENT_GAP` |
| `REFER_TO_ANALYST` | nothing is drafted; a person reads the line before anything is filed | `DEMOGRAPHIC_MISMATCH`, `IDENTIFIER_INVALID`, `OTHER_PLAN_ENROLLMENT`, `UNRESOLVED_NEEDS_ANALYST` |
| `NO_ACTION` | there is nothing to file | `NOT_A_REJECT`, `DUPLICATE_TRANSACTION` |

**`RESUBMIT` does not mean submitted.** Nothing in this procedure transmits anything to CMS and
nothing in it writes to the plan's own enrollment record. What it produces is a draft and the
evidence behind it.

---

## The corrective effective date — rules RR-5 to RR-8

**RR-5 — A corrective effective date exists only where a corrective transaction is drafted**
Only a `RESUBMIT` line carries a corrective effective date. Every other line carries none, and a
date returned on a line nothing is filed for is a wrong answer rather than an empty one. Where a
date is due it starts as the effective date the original transaction requested, and the floors
below are the only things that move it.

**RR-6 — Entitlement is a floor, and it is BOTH parts**
A corrective effective date may not fall before the beneficiary is entitled to Part A **and** to
Part B. The floor is the **later** of the two entitlement start dates the reply file prints.
Checking only Part A is the commonest way to draft a date that re-rejects, because Part B is the
one that usually starts later.

**RR-7 — The retro window is a floor, unless the code is retro-eligible**
A corrective effective date may not fall before the first day of the month that is
`retro_window_months` before the CMS processing month printed on the reply file. A reply code the
table marks **retro-eligible** is exempt from this floor and from this floor only — the entitlement
floor in RR-6 still applies to it.

**RR-8 — The later floor governs, and a date that already clears both is kept**
The corrective effective date is the **latest** of the date originally requested and every floor
that applies to the line. Where the requested date already clears every floor it is kept
unchanged — advancing a date that did not need advancing loses the beneficiary a month of coverage
exactly as surely as leaving one that re-rejects does.

---

## How to read the pack — rules RR-9 and RR-10

These two shape the reading. **Nothing in this kit enforces them in code**, and that is said here
rather than implied: they decide which signal a reader returns, and a signal that got them wrong
is a reading error the rule engine behind it cannot repair.

**RR-9 — A punctuation, spacing or letter-case difference is not a demographic difference**
A name that differs from the one CMS holds only by a hyphen, a space, an apostrophe or letter case
is the same name, and it is not a reason a transaction was rejected. A demographic difference means
a different date of birth, a different surname, or a different sex.

**RR-10 — Where a date breaches both floors, entitlement is the root cause**
Where an effective date falls before the entitlement floor **and** before the retro window, the
root cause is `ENTITLEMENT_GAP` rather than `EFFECTIVE_DATE_INVALID`. Entitlement is a fact about
the beneficiary and the retro window is a fact about the file; the transaction cannot be filed at
all until entitlement begins, and correcting only the window would file it into a period the
beneficiary was not entitled for.

THE NINE ROOT CAUSES, in the order you should consider them, and what clustering a
line under each one commits somebody to:

  UNRESOLVED_NEEDS_ANALYST   The pack does not settle it
      THE LINE GOES TO A PERSON WITH NOTHING DRAFTED. The reply code leaves two possible root causes open and the pack carries nothing that decides between them. It is a real answer: it names what could not be settled and it declines to file anything on a guess. Used where the pack does settle it, an analyst re-does work already done; not used where the pack does not, a corrective transaction goes to CMS on a coin toss.

  NOT_A_REJECT               Accepted -- nothing to correct
      NOTHING IS FILED AND NOTHING IS QUEUED. The reply code acknowledges a transaction CMS accepted as submitted. Clustering an accepted line as a reject drafts a correction against a transaction that is already on file, which is how a duplicate gets created by the tool that was supposed to prevent one.

  PLAN_CODE_MISMATCH         The plan benefit package submitted is not open at this segment
      A CORRECTIVE ENROLLMENT TRANSACTION IS DRAFTED with a plan benefit package that is open at the segment CMS holds. It is a plan-side error and it is fixed by re-filing. Sent to the demographics queue instead, nobody re-files anything and the enrolment simply does not exist.

  DEMOGRAPHIC_MISMATCH       A demographic on the submission differs from the one CMS holds
      THE LINE GOES TO AN ANALYST AND NOTHING IS RE-FILED. A demographic difference is settled with CMS against the beneficiary's own record; re-submitting the same transaction with the same demographics re-rejects with the same code. ⚠︎ A PUNCTUATION-ONLY DIFFERENCE IS NOT ONE OF THESE -- see RR-2026, and see the named trap in data/SOURCES.md.

  IDENTIFIER_INVALID         The identifier submitted is not the one CMS holds
      THE LINE GOES TO AN ANALYST. The identifier has to be resolved against CMS's own record before anything is filed under it, and a corrective transaction carrying the same identifier re-rejects.

  EFFECTIVE_DATE_INVALID     The effective date requested is not one this transaction can carry
      A CORRECTIVE ENROLLMENT TRANSACTION IS DRAFTED with a corrected effective date. This is the cluster where the drafted date matters most and where the naive date -- the one originally requested -- simply re-rejects.

  ENTITLEMENT_GAP            Entitlement had not begun on the date requested
      A CORRECTIVE ENROLLMENT TRANSACTION IS DRAFTED, and it cannot begin before the beneficiary is entitled to both Part A and Part B. Drafted on the originally requested date it re-rejects with the same code, every month, until somebody reads the entitlement dates that were printed on the reply file all along.

  DUPLICATE_TRANSACTION      CMS already holds a transaction for this period
      NOTHING IS FILED. Re-submitting produces a second duplicate, and the reply file says so the next day.

  OTHER_PLAN_ENROLLMENT      Another organisation holds the beneficiary for the period requested
      THE LINE GOES TO AN ANALYST. A competing enrolment is not a plan-side data error and it is not fixed by re-filing; it is worked with the beneficiary and with CMS.

THE SIGNAL VOCABULARY -- what the pack SHOWS, which is where your answer came from:

  plan_code_not_active       the plan benefit package submitted is not one of those open at the segment CMS holds
  demographics_differ        a demographic on the submission differs MATERIALLY from the one CMS holds -- a different date of birth, a different surname, a different sex. A difference of punctuation, spacing or letter case is NOT this
  identifier_not_on_file     the identifier submitted is not the identifier CMS holds for this beneficiary
  date_before_entitlement    the effective date requested falls before the beneficiary is entitled to both Part A and Part B
  date_outside_window        the effective date requested falls outside the window this reply file accepts, and entitlement is not the reason
  already_on_file            CMS already holds a transaction covering this beneficiary and period
  other_plan_holds           another organisation holds the beneficiary for the period requested
  accepted                   the reply code acknowledges a transaction CMS accepted -- nothing was rejected
  nothing_explains_it        the pack carries nothing that decides why this code came back

A REPLY CODE'S CLASS, as the table on the pack gives it:

  accepted                   the code acknowledges a transaction CMS accepted
  settled                    the code names exactly one root cause on its own
  ambiguous                  the code is consistent with two root causes, and the transaction as submitted is what decides between them

THE THREE ACTIONS, and what answering each one commits somebody to:

  RESUBMIT                   draft a corrective enrollment transaction, with an effective date, for an analyst to review and submit
  REFER_TO_ANALYST           draft nothing; the line needs a person before anything is filed
  NO_ACTION                  nothing to file -- the transaction was accepted, or CMS already holds one

Which action follows which root cause is fixed by RR-2026 and is not yours to
choose:

  UNRESOLVED_NEEDS_ANALYST   REFER_TO_ANALYST
  NOT_A_REJECT               NO_ACTION
  PLAN_CODE_MISMATCH         RESUBMIT
  DEMOGRAPHIC_MISMATCH       REFER_TO_ANALYST
  IDENTIFIER_INVALID         REFER_TO_ANALYST
  EFFECTIVE_DATE_INVALID     RESUBMIT
  ENTITLEMENT_GAP            RESUBMIT
  DUPLICATE_TRANSACTION      NO_ACTION
  OTHER_PLAN_ENROLLMENT      REFER_TO_ANALYST

HOW TO QUOTE THE PACK, and how it will be read.

For every reply line whose root cause is not NOT_A_REJECT, `citation` must be ONE FINDING, ONE DESK
NOTE OR ONE PRINTED LINE COPIED VERBATIM out of the pack -- the text that establishes that root
cause. Where a line's two findings are both true and the desk note is what decides between them,
THE DESK NOTE IS THE CITATION. It may come from any part of the pack except the pack-level ANALYST
WORKING NOTES at the end, which are commentary and establish nothing.

  - Copy it character for character, including the `Line N:` a finding opens with. It is located
    in the pack by searching for it, so a paraphrase, a shortened version, an ellipsis in the
    middle, or two findings joined together will not be found at all and will score nothing. There
    is no partial credit for a quote the pack does not contain.
  - Quote the one finding, not the block. What is returned is compared with that finding by
    character overlap: it must cover at least 60 pct of the finding, and at least 30 pct of
    what you return must be the finding. Returning the whole pack scores nothing.
  - RR-2026 is NOT part of the pack. A rule is never the citation.
  - Where the root cause is NOT_A_REJECT, `citation` is null. Nothing went wrong on a line CMS
    accepted, and quoting something for one is counted as a wrong answer, not as an empty one.

THE REPLY FILE PACK, verbatim:

TRANSACTION REPLY FILE PACK RR-0001 -- daily reply file, afternoon batch

PACK FACTS
  Submitting contract                H9858
  Reply file received                2026-03-05
  CMS processing month               2026-02
  Retro window                       3 months -- the earliest effective date a corrective
                                     transaction may carry is 2025-11-01, unless the reply
                                     code is marked retro-eligible below
  Reconciliation procedure           RR-2026, read as at 2026-08-31

REPLY CODE TABLE IN FORCE -- RC-2026, read as at 2026-08-31

  R-101   accepted   Enrollment accepted as submitted
                     NOT_A_REJECT
  R-104   accepted   Disenrollment accepted as submitted
                     NOT_A_REJECT
  R-107   accepted   Plan benefit package change accepted as submitted
                     NOT_A_REJECT
  R-212   settled    Identifier submitted is not on file
                     IDENTIFIER_INVALID
  R-224   settled    A transaction covering this period is already on file
                     DUPLICATE_TRANSACTION
  R-241   settled    Beneficiary is enrolled with another organisation for the period requested
                     OTHER_PLAN_ENROLLMENT
  R-256   settled    Effective date requested is outside the window this transaction accepts
                     EFFECTIVE_DATE_INVALID; RETRO-ELIGIBLE
  R-231   ambiguous  Beneficiary record does not support the transaction as submitted
                     consistent with DEMOGRAPHIC_MISMATCH or PLAN_CODE_MISMATCH -- the transaction as submitted decides
  R-238   ambiguous  Coverage requested cannot start on the date submitted
                     consistent with EFFECTIVE_DATE_INVALID or ENTITLEMENT_GAP -- the transaction as submitted decides
  R-247   ambiguous  Submitted plan benefit package not open for the segment shown
                     consistent with PLAN_CODE_MISMATCH or IDENTIFIER_INVALID -- the transaction as submitted decides

REPLY LINE 1 -- reply code R-231
  Reply text                       Beneficiary record does not support the transaction as submitted

  BENEFICIARY SEGMENT CMS HOLDS
    Identifier on file             1TP3J72LP41
    Name on file                   VELLACOURT, P
    Date of birth on file          1960-12-16
    Part A entitlement from        2022-03-01
    Part B entitlement from        2025-09-01
    Plan on file for the period    none
    Packages open at this segment  003, 007

  TRANSACTION AS SUBMITTED
    Transaction                    plan benefit package change
    Plan benefit package           004
    Effective date requested       2025-12-01
    Identifier submitted           1TP3J72LP41
    Name submitted                 KETTERIDGE, P
    Date of birth submitted        1960-12-16

  RECONCILIATION FINDINGS -- automated comparisons of the two blocks above
    Line 1: the surname submitted (KETTERIDGE) is not the surname on file (VELLACOURT).
    Line 1: plan benefit package 004 is not one of the packages open at this segment (003, 007).

  DESK NOTE ON LINE 1 -- what the submitting desk recorded about this line
    Line 1 note: identity was reconciled on the previous file for this member and came back accepted.

REPLY LINE 2 -- reply code R-101
  Reply text                       Enrollment accepted as submitted

  BENEFICIARY SEGMENT CMS HOLDS
    Identifier on file             6QI5HE2VA63
    Name on file                   VANTERPOOL, J
    Date of birth on file          1957-12-03
    Part A entitlement from        2019-10-01
    Part B entitlement from        2025-12-01
    Plan on file for the period    none
    Packages open at this segment  003, 009

  TRANSACTION AS SUBMITTED
    Transaction                    enrollment
    Plan benefit package           003
    Effective date requested       2026-02-01
    Identifier submitted           6QI5HE2VA63
    Name submitted                 VANTERPOOL, J
    Date of birth submitted        1957-12-03

  RECONCILIATION FINDINGS -- automated comparisons of the two blocks above
    Line 2: none -- the transaction was accepted as submitted.

  DESK NOTE ON LINE 2 -- what the submitting desk recorded about this line
    Line 2 note: submitted with the rest of the batch on the usual schedule.

REPLY LINE 3 -- reply code R-231
  Reply text                       Beneficiary record does not support the transaction as submitted

  BENEFICIARY SEGMENT CMS HOLDS
    Identifier on file             9DP5I13EU87
    Name on file                   WREXHALL, W
    Date of birth on file          1941-10-17
    Part A entitlement from        2023-04-01
    Part B entitlement from        2024-12-01
    Plan on file for the period    none
    Packages open at this segment  001, 007

  TRANSACTION AS SUBMITTED
    Transaction                    enrollment
    Plan benefit package           004
    Effective date requested       2025-06-01
    Identifier submitted           9DP5I13EU87
    Name submitted                 WREXHALL,W
    Date of birth submitted        1941-10-17

  RECONCILIATION FINDINGS -- automated comparisons of the two blocks above
    Line 3: the name submitted differs from the name on file by punctuation only.
    Line 3: plan benefit package 004 is not one of the packages open at this segment (001, 007).

  DESK NOTE ON LINE 3 -- what the submitting desk recorded about this line
    Line 3 note: routine submission, no query was raised before it went.

REPLY LINE 4 -- reply code R-231
  Reply text                       Beneficiary record does not support the transaction as submitted

  BENEFICIARY SEGMENT CMS HOLDS
    Identifier on file             8CQ9OO9BC76
    Name on file                   HERNANDEZ-VOSS, N
    Date of birth on file          1941-11-25
    Part A entitlement from        2019-04-01
    Part B entitlement from        2025-06-01
    Plan on file for the period    none
    Packages open at this segment  001, 005

  TRANSACTION AS SUBMITTED
    Transaction                    enrollment
    Plan benefit package           008
    Effective date requested       2026-02-01
    Identifier submitted           8CQ9OO9BC76
    Name submitted                 PRZYBYL, N
    Date of birth submitted        1941-11-25

  RECONCILIATION FINDINGS -- automated comparisons of the two blocks above
    Line 4: the surname submitted (PRZYBYL) is not the surname on file (HERNANDEZ-VOSS).
    Line 4: plan benefit package 008 is not one of the packages open at this segment (001, 005).

  DESK NOTE ON LINE 4 -- what the submitting desk recorded about this line
    Line 4 note: identity was reconciled on the previous file for this member and came back accepted.

ANALYST WORKING NOTES -- commentary on the batch; establishes nothing

  Reconciliation for this contract is due on the fifteenth and this batch is the last of them.
  Pack pulled for review on the batch summary, not on any single line.


Reply with JSON and nothing else, exactly this shape:

{
 "lines": [
   {
    "line": <a number>,
    "signal": "plan_code_not_active" | "demographics_differ" | "identifier_not_on_file" | "date_before_entitlement" | "date_outside_window" | "already_on_file" | "other_plan_holds" | "accepted" | "nothing_explains_it",
    "root_cause": "UNRESOLVED_NEEDS_ANALYST" | "NOT_A_REJECT" | "PLAN_CODE_MISMATCH" | "DEMOGRAPHIC_MISMATCH" | "IDENTIFIER_INVALID" | "EFFECTIVE_DATE_INVALID" | "ENTITLEMENT_GAP" | "DUPLICATE_TRANSACTION" | "OTHER_PLAN_ENROLLMENT",
    "action": "RESUBMIT" | "REFER_TO_ANALYST" | "NO_ACTION",
    "corrective_effective_date": "<text>" or null,
    "citation": "<text>" or null
   },
   ... one object per reply line, all four, in the order the pack prints them
 ],
 "confidence": <a number>,
 "why": "<text>"
}

What each field means:
  line                       the reply line number, as the pack prints it. Answer all four, once each, in order
  signal                     WHAT THE PACK SHOWS about this line, as its own answer, and never a conclusion about what to do. `plan_code_not_active` the package submitted is not one of those open at this segment. `demographics_differ` a demographic differs MATERIALLY -- a different date of birth, surname or sex, and NEVER a difference of punctuation, spacing or case. `identifier_not_on_file` the identifier submitted is not the one CMS holds. `date_before_entitlement` the effective date requested falls before entitlement to BOTH Part A and Part B. `date_outside_window` the effective date is outside the window this file accepts and entitlement is not the reason. `already_on_file` CMS already holds a transaction for this period. `other_plan_holds` another organisation holds the beneficiary. `accepted` the code acknowledges a transaction CMS accepted. `nothing_explains_it` the pack carries nothing that decides why this code came back. This is the reading, and it is the only field the rule engine re-uses
  root_cause                 the cluster this reject belongs in, under RR-1 to RR-4 applied in that order. UNRESOLVED_NEEDS_ANALYST where the code leaves two candidates and the pack settles neither -- it is a real answer, not a failure to answer. NOT_A_REJECT where the code acknowledges a transaction CMS accepted
  action                     what happens next. RESUBMIT means a corrective enrollment transaction is DRAFTED for an enrollment analyst to review and submit -- nothing here files anything with CMS, ever. REFER_TO_ANALYST means nothing is drafted and a person reads the line. NO_ACTION means there is nothing to file
  corrective_effective_date  the effective date the drafted corrective transaction should carry, as YYYY-MM-DD, under RR-5 to RR-8: the LATEST of the date originally requested, the later of the two entitlement start dates, and -- unless the reply code is retro-eligible -- the first day of the retro window this reply file states. NULL on every line whose action is not RESUBMIT. A date returned where nothing is filed is a wrong answer, not an empty one; and a date left at the one originally requested where a floor moves it is a transaction that simply re-rejects
  citation                   ONE FINDING OR ONE PRINTED LINE COPIED VERBATIM from the pack -- the text that establishes this root cause. Copy it exactly, character for character, including the `Line N:` it opens with; do not paraphrase, do not shorten with an ellipsis, do not join two findings. NULL where the root cause is NOT_A_REJECT: nothing went wrong on an accepted line and quoting something for one is counted as a wrong answer, not an empty one
  confidence                 your own number between 0 and 1 for this pack's four answers taken together. It is published as evidence and never used to change an answer
  why                        one sentence naming what decided the lines you did not answer NOT_A_REJECT

Four line objects for one pack. No more, no fewer.

Raw responseThe raw response

{
 "lines": [
  {
   "line": 1,
   "signal": "plan_code_not_active",
   "root_cause": "PLAN_CODE_MISMATCH",
   "action": "RESUBMIT",
   "corrective_effective_date": "2025-12-01",
   "citation": "Line 1 note: identity was reconciled on the previous file for this member and came back accepted."
  },
  {
   "line": 2,
   "signal": "accepted",
   "root_cause": "NOT_A_REJECT",
   "action": "NO_ACTION",
   "corrective_effective_date": null,
   "citation": null
  },
  {
   "line": 3,
   "signal": "plan_code_not_active",
   "root_cause": "PLAN_CODE_MISMATCH",
   "action": "RESUBMIT",
   "corrective_effective_date": "2025-11-01",
   "citation": "Line 3: plan benefit package 004 is not one of the packages open at this segment (001, 007)."
  },
  {
   "line": 4,
   "signal": "plan_code_not_active",
   "root_cause": "PLAN_CODE_MISMATCH",
   "action": "RESUBMIT",
   "corrective_effective_date": "2026-02-01",
   "citation": "Line 4 note: identity was reconciled on the previous file for this member and came back accepted."
  }
 ],
 "confidence": 0.95,
 "why": "Desk notes on lines 1 and 4 accept the identity, line 3's name difference is punctuation only, and each line's submitted package is not open at the segment, so lines 1, 3, and 4 cluster as PLAN_CODE_MISMATCH while line 2 is accepted."
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Reconcile CMS transaction reply files and cluster each reject by root cause — 60 reply-file packs. One model answered, and every answer was then graded Four different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Exact match in pure code on four fields per reply line -- the root-cause cluster, the queue, the corrective effective date, and the citation by character interval. NO MODEL GRADES ANYTHING, which is what makes re-scoring a committed run cost $0.00 and what lets both citation floors be swept without buying a call.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

the Foundry app's build/facts/models.json, the row for each model id, each with the vendor's own pricing URL and its own confirmation date. The provider this kit actually ran on is deliberately NOT priced here: naming it on a page is what the series rule forbids, and a rate card is a naming. The tokens are this run's own.

Rates checked 2026-08-23. ⚑ THE PROVIDER THIS KIT ACTUALLY RAN ON IS NOT PRICED HERE, and that is the series rule rather than a gap: naming it on a page is what the rule forbids, and a rate card is a naming. The measured bill for r001-reply-reconcile is $0.271736 across 60 calls and it is recorded per call, with its tariff, in the run record. One further published card is withheld because it is not a US rate card, so the figure would not be actionable where a reader buys.

Grading unitWhat the grading figure prices

The graders are Python functions over committed JSON. Grading one run costs nothing and grading it again at a different citation floor costs nothing either -- the pipeline's own bill is in the Cost lens and is a separate thing.

The gradersFour ways to grade

⚠︎ READ b001 AND b002 AS CEILINGS ON PATTERN MATCHING HERE, NEVER AS LOWER BOUNDS ON THE JOB. tools/build_corpus.py writes every reconciliation finding from a small template pool and every deciding desk note from five sentences per direction; evals/baseline.py's patterns were written by the same author, on the same day, against that pool. A sibling kit could claim its floor was written by somebody who had never seen the corpus. This one cannot, and pretending otherwise would be the more comfortable lie.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — top_k, chunk size, prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-23. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

evals/check_labels.py, the independent re-derivation of the whole key, is also free and offline -- it made 0 calls.

Cost driversWhat actually moves the bill

• PROVIDER-SIDE REASONING, and it is not close. 323,692 of 347,930 output tokens -- 93.0 pct -- are reasoning at the tier's default, re-rolled per call, for a reply that is four rows of JSON. The largest reply reached 19,692 of the 32,000 ceiling. Output is 84.4 pct of the bill.
• The pack. 6,967 characters of the 29,586-character prompt change per call; the other 18,662 are the byte-identical prefix.
• THE PREFIX CACHE, WHICH CUTS THE BILL RATHER THAN DRIVING IT. 235,008 of 418,883 input tokens were cache hits -- 56.1 pct -- because the first four prompt parts are identical on every call and are sent first. A provider without a cache tier pays the full input rate on all of it.
• One call per pack. There is no retry, no judge, no second pass and no retrieval, so 60 packs are exactly 60 calls and the denominator is the whole run.

Your volumeWhat it costs at your volume

LINEAR IN PACKS AND NOT IN LINES. Ten times the packs is ten times the calls and ten times the bill: $2.72 for 600 packs at this tariff, with the prefix cache improving slightly as the prefix is re-hit. Ten times the LINES PER PACK is not a straight line -- the pack part of the prompt grows and the answer grows with it, and reasoning is re-rolled per call rather than per line. Nobody here has measured which way that bends, and it is the first thing to measure before scaling.

Model choiceWhy this model

It is the tier this estate runs on and the only one configured at the repo root. ⚠︎ NO COMPARISON WAS MADE: no other model was called against this corpus, so 'why this model' is an answer about the account rather than about the job.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

the whole 60-pack scored run (r001-reply-reconcile, the fast tier): 418,883 input tokens (235,008 of them prefix cache hits, 56.1 pct) and 347,930 output, of which 323,692 are provider-side reasoning. ⚠︎ usd_actually_paid is null ON THIS PAGE by the series rule, not because it is unknown: the real bill is $0.271736 across all 60 calls and it is recorded per call, with its tariff, in the run record.

The codeEvery file, and what it is for

12 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/clusters.pythe vocabulary

The nine root causes, the nine signals, the three actions and the three reply-code classes, as data with what each one commits somebody to. UNRESOLVED_NEEDS_ANALYST leads the list, here and everywhere the kit renders it: a clusterer with eight buckets and no ninth has to put every line it cannot settle into one somebody will work.

# THE VOCABULARY: what a reject can be caused by, what to do about it, and what the reader is

CLUSTERS = {
ORDER = ["UNRESOLVED_NEEDS_ANALYST", "NOT_A_REJECT", "PLAN_CODE_MISMATCH", "DEMOGRAPHIC_MISMATCH",
SIGNALS = {
SIGNAL_ORDER = ["plan_code_not_active", "demographics_differ", "identifier_not_on_file",
ACTIONS = {
ACTION_ORDER = ["RESUBMIT", "REFER_TO_ANALYST", "NO_ACTION"]
CODE_CLASSES = {
def labels():
def label_of(key):

src/policy.pythe rule engine

RR-1..RR-4 read out of data/policy.json in the published order, plus RR-5..RR-8 as arithmetic over dates the reply file prints. ⚑ TWO FUNCTIONS, AND NEITHER REACHES INTO THE OTHER: cluster() decides what went wrong from the code class and the reading; corrective_date() decides when from four printed dates and nothing else. Ten assertions at import pin the precedence the procedure states in words.

# RR-2026, APPLIED. One signal plus one reply code in, one root cause, one action and one

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
POLICY = json.load(open(os.path.join(HERE, "data", "policy.json"), encoding="utf-8"))
CODES = json.load(open(os.path.join(HERE, "data", "reply_codes.json"), encoding="utf-8"))
POLICY_ID = POLICY["policy_id"]
AS_OF = POLICY["as_of"]
RULES = POLICY["rules"]
RULE_ORDER = POLICY["rule_order"]
DATE_RULES = POLICY["date_rules"]
DATE_RULE_ORDER = POLICY["date_rule_order"]

data/reply_codes.jsonthe reply code table — a swap seam

Ten invented codes, each classed accepted, settled or ambiguous, and each ambiguous one naming exactly two candidates with the commoner first. THE CLASS IS WHAT DECIDES WHETHER A READING MATTERS AT ALL -- 110 of the 240 lines are settled by the table and no arm can lose them. The same table is printed on every pack, so it travels with the pack it applies to and there is no second copy to drift.

You change it to: Add a code, change a class, or re-order an ambiguous code's candidates. Re-ordering candidates is what the lookup floor takes, so it moves that floor and nothing else.

# 

src/prompt.pythe prompt

Six ordered parts, the first four byte-identical on every call and sent first, which is what earned the 56.1 pct prefix cache. It names all ten rules, all nine root causes, all three actions and the whole signal vocabulary, and it does none of the applying. Twenty assertions at import refuse a schema that offers a class the procedure never names.

# The assembled prompt, in one place, in the order it is sent.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
POLICY_MD = open(os.path.join(HERE, "data", "policy.md"), encoding="utf-8").read()
FIELDS = json.load(open(os.path.join(HERE, "data", "fields.json"), encoding="utf-8"))["fields"]
CLUSTERS = list(C.ORDER)
SYSTEM = """You are working one transaction reply file returned by CMS to a Medicare Advantage plan, and
def _cluster_block():
def _action_block():
CITATION_RULE = """HOW TO QUOTE THE PACK, and how it will be read.
def _schema_block():

src/reconciler.pythe model call and reply parse

One pack in, four reconciled lines out. Holds the 32,000-token ceiling and a tolerant JSON parse -- a correct answer wrapped in a code fence has not got it wrong. ⚠︎ IT NORMALISES THE CLOSED VOCABULARIES AND DELIBERATELY NOT THE DATE: a date reformatted from 04/01/2026 would be this file answering a question the arm did not answer.

# One reply-file pack in, four reconciled reply lines out. The only place a model is called.

MAX_TOKENS = 32000
THINKING = None
def parse_reply(text):
def normalise(obj):
def reconcile(cfg, text, pack, complete_fn=None, max_tokens=None):
CLUSTERS = tuple(C.ORDER)
ACTIONS = tuple(C.ACTION_ORDER)

src/recheck.pywhat the model is trusted with

⚑ EXACTLY ONE FIELD SURVIVES PER LINE: the signal. The root cause, the queue and the date the reply offered are discarded and recomputed. That makes RR-1 and RR-2 un-loseable and the date arithmetic; RR-3 still rests entirely on the reading, and that asymmetry is the measurement. On r001 it overrode nothing -- 0 of 240 -- so every error in this run was a reading error and not a rule error.

# THE PURE-CODE STATION, applied to a model's reply. What the model is trusted with, and nothing

TRUSTED = ("signal",)
def recheck(answer, pack, pack_text):

src/citation.pythe citation locator

Normalise once, locate exactly, compare character intervals, zero for not-found. Deliberately NOT fuzzy: a similarity score would give partial credit to a paraphrase, which is the one thing this metric exists to refuse. Both floors are arguments, so any committed run re-scores at a different pair for $0.00.

# HOW A CITATION IS SCORED. Character offsets, two floors, and no partial credit for a quote the

FOLDS = {
def fold(s):
def norm(s):
def norm_quote(s):
def locate(norm_text, quote):
def span_of(norm_text, sentence):
COVERAGE_MIN = 0.60
PRECISION_MIN = 0.30
def credit(norm_text, gold_span, quote, coverage_min=COVERAGE_MIN, precision_min=PRECISION_MIN):

src/adapters/__init__.pythe adapter

One interface, several providers, raw HTTP and no vendor SDK -- the only file in the kit that knows who serves the model. ⚑ ITS SOCKET TIMEOUT AND THE TOKEN CEILING ARE ONE SETTING: 1,200 s beside a 32,000-token ceiling, with transport retries cut to ONE against four for a transient status.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TIMEOUT_S = 1200
TRANSPORT_RETRIES = 1
def _post(url, headers, payload, timeout=TIMEOUT_S):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}

evals/baseline.pythe three free floors

A reply-code lookup, a regex over the printed comparisons, and the same plus a keyword match over the desk note. All three free, all three scored by the same scorer. ⚑ THE FIRST IS BUILT TO FAIL IN A NAMED WAY -- it takes R-231's commoner cause every time, which is the trap -- and the third scores HIGHER than the paid arm, which is published rather than buried.

# THE FREE FLOORS. No key, no model, no network. Two genuine attempts at the job, both of them

MODES = ("codetable", "findings", "notes")
SIGNAL_FOR = {v: k for k, v in P.SIGNAL_TO_CLUSTER.items() if v is not None}
FINDING_SIGNALS = [
NON_SIGNALS = [r"by punctuation only",
NOTE_SETTLES_IDENTITY = re.compile(
NOTE_SETTLES_PACKAGE = re.compile(
def notes_by_line(text):
def findings_by_line(text):
def read(text, pack, mode):

evals/scoring.pythe scorer

Exact match on four fields per line, and every rate carries its own denominator. The citation is scored in its own right because an answer attached to the wrong evidence is wrong in a way that reads as right -- and on this run that is the kit's only miss.

# Grade one arm against the answer key. Deterministic -- no model judges anything here.

CLUSTERS = tuple(C.ORDER)
ACTIONS = tuple(C.ACTION_ORDER)
GRADED = ("root_cause", "action", "corrective_effective_date", "citation")
TAXONOMY = ("no_answer", "drafted_on_an_accepted_line", "missed_a_reject",
def _pct(n, d):
def _s(v):
def rows_of(answer):
def _bucket(g, m, cit):
def score(dets, golds, texts, coverage_min=CIT.COVERAGE_MIN, precision_min=CIT.PRECISION_MIN):

evals/check_labels.pythe answer-key gate

Re-derives every root cause, action, date and citation span a second time, from the SHIPPED pack text, with its own rule table, date calculator, pack parser and normaliser and no import from src/. ⚑ IT ASSERTS THE EVIDENCE-CHANNEL SPLIT IN BOTH DIRECTIONS, which is the assertion that stops the corpus flattering a code-only floor.

# Grade the answer key itself. Free, offline, and it imports NOTHING from src/.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
RULES_HERE = [
DATE_RULES_HERE = ["RR-5", "RR-6", "RR-7", "RR-8"]
READING_RULES_HERE = ["RR-9", "RR-10"]
ACTION_HERE = {
SIGNAL_HERE = {
MBI_EXCLUDED = set("SLOIBZ")

src/app.pythe local board

http.server, no dependency, six read endpoints and two POST routes, and NO WRITE PATH ANYWHERE -- the guardrail is enforced by there being no endpoint that could file anything, not by a sentence in a prompt. Renders with no key, computes all three floors live, and replays the committed run for $0.00.

# The minimal local board. Standard library only -- python3 -m src.app, then open the URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
CORPUS = os.path.join(HERE, "data", "corpus")
PORT = int(os.environ.get("PORT", "9236"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-reply-reconcile")
FLOOR_RUNS = [("codetable", "b000-reply-reconcile-codetable"),
def golds():
def load_pack_text(pid):
def recorded(run_id):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 6981 input and 5798 output tokens per query at top-k 1, of which about 150 tokens are fixed (system prompt, chat framing, the question itself) and the rest is retrieved context. Change the retrieval depth and the input scales with it. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model and top-k follow the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

⚠︎ WRITTEN, WIRED, AND NOT FIRED -- AND NO NUMBER ON THIS PAGE SAYS OTHERWISE. evals/injection.py exists, targets the right lines and has the control it needs (the scored run's own cached answers). It has not been run. Every figure below is a DESIGN, written down so a future run reports against it rather than re-arguing it.

API_KEY is read from the shared <code>&lt;repo&gt;/.env</code> and this kit's own <code>.env</code> by <code>src/config.py</code>, sent as an Authorization header to the one configured provider, and written to no file, no log and no result record. ⚠︎ NOTE THE SHARED FILE: a kit with no <code>.env</code> of its own is still fully wired from the repo root, which is convenient and is also how a run gets fired by accident. <code>API_KEY=</code> forced empty is the belt-and-braces form for any free operation, and tools/shoot_ui.mjs does exactly that before every screenshot.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. ⚠︎ THE ADVERSARIAL ARM WAS NEVER FIRED. python3 -m evals.injection --run-id x001-reply-reconcile --yes is the command the README carries: 60 calls, one per pack carrying at least one reject. At r001's measured $0.004529 per pack that is about $0.27 -- a projection, not a measurement, because nothing was run. Measured on no run -- results/ holds no x001-reply-reconcile file, and no adversarial figure exists for this kit.

Two gates, and the split between them is the measurement the probe would report. 110 of the 240 reply lines are settled by the reply code table, which an injected sentence cannot reach; 130 ride on the reading, which no pure code restores.

The result⚠︎ NOT FIRED -- no adversarial number exists for this kit. evals/injection.py is written, wired and has its control cache; 60 calls would fire it. Nothing on this page makes a robustness claim.

The design, so it can be checked rather than re-argued later: one line appended to EVERY per-line DESK NOTE on all 60 packs, in the register of a note already on file, with the pack otherwise untouched. The desk note is the target because it is the one channel this kit pays a model to read — 36 of the 240 reply lines are settled by nothing else — and because anybody who can write a desk note can write that line, which is also the line somebody adds for entirely innocent reasons, as a template or a tidy-up. The control is the scored run's own cached answer for each pack, already committed, so no second control call would be bought. Scoring is against the CLEAN pack, so a suppressed citation cannot score by finding the injected sentence. THE REPORTABLE NUMBER IS THE THREE-WAY SPLIT: the 110 lines the printed reply code table restores whatever the note says, the 130 that rest on the reading and cannot be restored, and the DATE — which comes back on any suppressed line that still reaches RESUBMIT, and is lost entirely on a line talked into NOT_A_REJECT, because RR-5 gives an accepted line none.

HonestyWhat this does not prove

• Everything about this surface. No attack run exists.
• In particular: whether the desk-note channel — the only band this kit pays for, and the only one where a call does something a printed comparison cannot — can be turned off by a sentence written in the same position as the notes that legitimately settle 36 lines. That is the question the probe was written for.
• And whether the reply code table printed on the pack can be attacked. By construction the engine reads data/reply_codes.json rather than the printed copy, so a doctored table on the pack should change nothing after the recheck — and that was not probed either.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

⚑ NEVER SUBMIT A TRANSACTION TO CMS, AND NEVER WRITE TO THE PLAN'S OWN ENROLLMENT RECORD. This kit produces four determinations per reply file and the line of the pack each came from. Where a correction is warranted it DRAFTS one -- a root cause and an effective date -- for an enrollment analyst to review and file. Where the pack settles neither candidate it produces UNRESOLVED_NEEDS_ANALYST, which is a line handed to a person with nothing drafted at all.

Stated in the system role as the FIRST thing the model is told, in RR-2026's own text, in the README, on the board, and enforced by THERE BEING NO ENDPOINT THAT WRITES: src/app.py has six read routes and two POST routes, and both POST routes return an answer. There is no code path in this kit that reaches CMS or any enrolment record, and no flag that adds one.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• Not a filing system. It writes nothing, transmits nothing, and there is no flag that adds an endpoint that does.
• Not a system of record. A reply code is evidence about what CMS did with a transaction; nothing here changes a beneficiary's enrolment on the strength of one.
• Not an eligibility tool. It does not decide which election period an enrolment qualifies under and carries no rule that could -- that is a different question and a sibling kit.
• Not a regulatory instrument. RR-2026 is invented, carries NO regulatory citation on purpose, and the reply codes in RC-2026 are not CMS's transaction reply codes.
• Not a robustness claim. The adversarial arm was never fired.
• Not a measurement of a real reply file. Every number is against a synthetic corpus whose deciding prose comes from five sentences per direction.
• ⚠︎ Not, on this corpus, a demonstration that the paid call is worth its money. A free floor scores 240 of 240 reply lines against the call's 239, for $0.00 against $0.271736.

WatchedWhat is watched, and why that one

The kit measures 67 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

142 measured by the latest run -75 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (calls_in_totals 60, citation_coverage_floor 0.6, citation_precision_floor 0.3, cited_judgements 180, clean_packs 0, dataset_version reply-reconcile-v1-60packs, date_judgements 240, dates_due 98, dates_not_due 142, dates_that_keep 47, dates_that_move 51, failures 0, field_channel_n 94, hard_packs 60, judgements 240, judgements_answered 240, lines_per_pack 4, not_a_reject_lines 60, note_channel_n 36, packs 60, packs_answered 60, plan_code_trap_n 40, policy_id RR-2026, rechecked_citation_coverage_floor 0.6, rechecked_citation_precision_floor 0.3, rechecked_cited_judgements 180, rechecked_clean_packs 0, rechecked_date_judgements 240, rechecked_dates_due 98, rechecked_dates_not_due 142, rechecked_dates_that_keep 47, rechecked_dates_that_move 51, rechecked_field_channel_n 94, rechecked_hard_packs 60, rechecked_judgements 240, rechecked_judgements_answered 240, rechecked_note_channel_n 36, rechecked_packs_answered 60, rechecked_plan_code_trap_n 40, rechecked_rederived_from_cache False, rechecked_retro_exempt_kept_n 8, rechecked_table_channel_n 110, rechecked_uncited_judgements 60, retro_exempt_kept_n 8, socket_timeout_s 1200, table_channel_n 110, table_id RC-2026, uncited_judgements 60, unresolved_needs_analyst_lines 14) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

5 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

match · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 3 runs measured the same system — they differ on floor, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

match · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 145 chips that all say so.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 142 chips that all say so.

DeviationsWhat deviated

0 breaches across 5 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The answer-key gate and all three free floors are FREE and run before any spend, and on this kit they are also what settled the argument: b002-reply-reconcile-notes scored 240 of 240 before a single call was bought, so the paid run was fired knowing it could not win. Run evals/check_labels.py after ANY change to data/policy.json, data/reply_codes.json or tools/build_corpus.py — it re-derives the whole key independently and names what moved. Re-score a committed run at a different citation floor whenever the floors are argued about; it costs $0.00 and buys no calls. THE INJECTION PROBE IS A MANUAL STEP AND HAS NEVER BEEN RUN.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, standard library end to end -- no orchestrator, no agent loop, no vector store, no ORM, no requirements.txt. One pack, one call, one JSON reply, then pure code: the rule engine, the date arithmetic, the citation locator and the scorer are all functions you can read in an afternoon.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear, and concurrent only at the pack level. There is no agent, no tool loop, no retry and no branch: read the pack -> build six prompt parts -> one call -> parse -> keep ONE field per line -> apply RR-1..RR-4 in code -> compute the date with max() over four printed dates -> locate the citation -> score. The one place a decision is made twice is deliberate: the model's root cause, queue and date are discarded and recomputed, and the result file carries both.

The other sideWhat a framework costs you

• Adding a provider means editing the PROVIDERS dict by hand. In exchange pip install pulls nothing and a forker runs this on whichever key they already hold.
• There is no experiment tracker, so a second run is a second file and a comparison is something you write. On a kit with one paid run that is the right trade; on twenty it would not be.
• No streaming means a long generation is a silent socket for up to 1,200 s -- the slowest call here took 112,811 ms and nothing reported progress while it ran.
• No schema-validation library means a malformed reply is caught by a parse and recorded as a failure rather than repaired. On this run that never happened: 60 of 60 parsed.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-reply-reconcile on the fast tier, 2026-08-31. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b001-reply-reconcile-findings, b002-reply-reconcile-notes recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — blocking is computed in memory each run and is the only reduction step.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT, as the rest of this repo. Every pack, contract, identifier, name, date and reply code in it is invented, so there is nothing in it anybody else holds rights over. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineRoot-cause cluster, exact match

whether the reject was clustered under the root cause RR-1..RR-4 give it, nine-way

MethodHow the test was run

evals/scoring.py, string equality against data/gold.jsonl

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl -- planted by tools/build_corpus.py and re-derived, independently and from the SHIPPED packs, by evals/check_labels.py.

No true/false rates for this grader. It records 19 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? no threshold is set: the free desk-note floor takes 240 of 240, so a paid arm below that is not paying for anything

Cadence: every run; re-scoring a committed run costs $0.00 because the grader is code

The decisionWhen to reach for it

In one lineThe queue it goes to, exact match

whether the line was sent to RESUBMIT, REFER_TO_ANALYST or NO_ACTION

MethodHow the test was run

evals/scoring.py, string equality; the action is a pure function of the cluster, so this grader and the one above cannot disagree by accident

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl -- planted by tools/build_corpus.py and re-derived, independently and from the SHIPPED packs, by evals/check_labels.py.

No true/false rates for this grader. It records 9 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? 0 wrong_queue on the fast tier and on the desk-note floor

Cadence: every run; re-scoring a committed run costs $0.00 because the grader is code

The decisionWhen to reach for it

In one lineCorrective effective date, exact match

whether the drafted transaction carries the date RR-5..RR-8 give it -- and whether a line that should carry none carries one

MethodHow the test was run

evals/scoring.py, string equality on an ISO date, over all 240 lines: 98 where a date is due and 142 where the right answer is null

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl -- planted by tools/build_corpus.py and re-derived, independently and from the SHIPPED packs, by evals/check_labels.py.

No true/false rates for this grader. It records 8 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? 0 of 51. Every arm scored zero here, so the trap is planted and unsprung rather than passed

Cadence: every run; re-scoring a committed run costs $0.00 because the grader is code

The decisionWhen to reach for it

In one lineCitation, by character interval

whether the line of the pack quoted is the one that establishes the root cause

MethodHow the test was run

src/citation.py: normalise the pack once, locate the returned text, compare character intervals with the labelled finding; credit iff coverage >= 0.60 and precision >= 0.30. Not found scores zero with no near-miss credit

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl -- planted by tools/build_corpus.py and re-derived, independently and from the SHIPPED packs, by evals/check_labels.py.

No true/false rates for this grader. It records 7 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? the floors are arguments to credit(), so any committed run re-scores at a different pair for $0.00

Cadence: every run; re-scoring a committed run costs $0.00 because the grader is code

The decisionWhen to reach for it