The business caseThe problem this solves

A parent's note arrives -- an email, a portal message, a transcribed voicemail, a handwritten note -- and somebody in the front office reads it against the attendance policy before a record is confirmed: which dates are school days, how many in a row, did the note beat the third-school-day deadline, does this reason need a document and did the office actually log one. Most of that is arithmetic with a reading problem in front of it, and the reading is three sentences of informal prose that says 'Wednesday through Monday' across a weekend that also holds an early-release day. The front-office read of each excuse note against BP 5113 -- the date arithmetic, the deadline check and the documentation lookup -- before a person confirms the record.

The inputThe actual excuse notes

The corpus is 60 excuse notes, 0.02 MB (txt 60). A defect mix an attendance office would recognise and no public dataset can provide: ranges across weekends and a holiday, an early-release day inside a span, notes logged on the fourth school day, attachments claimed in the note and never logged by the office, vacations dressed as family obligations, two reasons in one note, relative dates, a name that does not match the student on record. Two thirds of the notes are planted because the clean case teaches nothing about the arithmetic -- and a third are clean because a corpus that is all traps measures a different job from the one the office has.

[Handwritten note on lined paper, transcribed by the front office. Received Thursday, September 17, 2026 at 7:57 PM.]

To whom it may concern,

Wesley Nguyen was out Sept 16 and 17: there was a death in the family and we were away for the services. Let me know if you need anything else.

Monica Nguyen

The outcomeWhat a good result looks like

A coded record a person confirms instead of a note a person deciphers: the code, the school days it covers, the documentation the policy demands and whether the office logged it, a verify flag when the note names the wrong student, and one sentence citing the clause.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Your notes are short, your dates are explicit, and your policy is mostly day-counts and deadlines — the free floor (evals/baseline.py) in front of the policy engine
  87.7 pct of records all-correct for $0.00, no key, no network, 0 false excuses -- and every miss it makes is a reading miss, so on genuinely simple notes it makes almost none. The engine, not the model, already applies every rule.
• Notes with two reasons, relative dates, dressed-up vacations and names to match -- the queue an office actually has — the model WITH the recheck -- the pair this kit ships
  The model beats the floor by FIVE RECORDS AS ANSWERED (62 of 65 against 57) and by EIGHT once both arms are rechecked (65 against 57) -- the floor runs the same recheck and gains nothing from it, so the whole eight is reading the keyword coder cannot do. The recheck turns 95.4 into 100.0 on this corpus without a second call, and takes false excuses from 3 to 0.
• You want to know whether any of this transfers to your district — your own corpus, through the same harness, before believing any number here
  The policy and calendar are data files; the harness, floor and scorer run keyless. Sixty of your own notes with a hand-checked key would tell you more than this page can.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-30. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-30 — r001-absence-code. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Measured on a copy of the kit with no .env and no API key, on this machine: python3 -m evals.check_labels re-graded the key (KEY CLEAN, 60 notes, 65 records, 21 cases) and python3 -m evals.run --floor rules scored the free floor, together in 0.11 s; python3 -m src.app served the board (HTTP 200) with the model button disabled and saying why. The corpus, the key, the policy, the calendar and both recorded model runs ship in the repo, so the whole product renders before anyone decides to spend.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one coding -- one note, one model call, end to end including provider-side reasoning tokens, on a shared connection. The recheck adds nothing measurable: it is pure Python over the reply. ⚠︎ NOT AN SLA: this credential is shared with sibling kits and these figures are a wall-clock observation. 60 notes ran in 218.6 wall seconds at 5 concurrent workers; the free floor and the policy engine answer all 60 in 0.11 s with no network at all.

Current processWhat it replaces

The front-office read of each excuse note against BP 5113 -- the date arithmetic, the deadline check and the documentation lookup -- before a person confirms the record.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• md
• 2 of 8 stations carry a recorded failure

scroll → 8 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one coded absence record is

No train/test split, because nothing is trained or tuned, and no segmentation, because one note goes whole into one call (the sizes here are the notes' own bytes). The unit every rate stands on is the CODED ABSENCE RECORD: 60 notes carry 65 of them, five notes splitting in two where a second reason or a policy rule codes days apart. 20 notes are clean; 40 each plant exactly one thing BP 5113 decides by arithmetic or by the office record -- 22 records are UNX or PENDING, 3 late notes, 3 partial days, 2 name mismatches, 21 named cases in all.

SetupWhat the setup figure measured

There is no index to build. The note, the policy text and a computed calendar window go whole into one prompt at 2,731.5 input tokens on average. The 0.1 s is what the whole free half costs: evals/check_labels.py re-grading the key plus the rules floor answering all 60 notes, measured end to end on a keyless copy of the kit.

LicenceLicence

MIT, with the corpus generated in-process -- there is no third-party data in this kit to licence.

Bring your ownBring your own excuse notes

Drop your own notes as .txt files into data/corpus/ and add one office-record row per note in data/notes.json -- the student on record, the received date, the attachments logged, any form approval date. Your district's numbers go in data/policy.json and your term in data/calendar.json; the engine, the floor and the recheck all read those files, so a policy change is a data change. The prompt reads data/policy.md, so keep the two in step -- evals/check_labels.py reads the JSON and the corpus, never the markdown, which is the honest limit of that check. Without a gold.jsonl of your own the UI works and the eval has nothing to score against.

⚠︎ And what stops being true when you do: ⚠︎ DO NOT point this at real notes casually. A real excuse note is protected information about a minor and it reaches your configured provider verbatim -- this kit has no redaction step, and the seam that protects documentation (the office record) protects nothing in the note's own text.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — but they are the input total apportioned by each part’s share of the characters, not a per-part reading from the provider’s counter. The total is measured; the split is arithmetic, and a part carrying denser text really costs more of it than this says.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 2,664 tokens assembled, 1,459 are rulebooks — 55% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The exact six parts sent for AN-0001, the first note of run r001-absence-code, replayed from the kit's own src/prompt.py build(). Trustworthy as the prompt that was sent because the replayed part names and character counts (1,571 / 5,287 / 930 / 343 / 324 / 1,201 = 9,656) match the decomposition the run itself recorded.

Full promptThe prompt in full

You are coding one parent's absence note for the attendance office of Riverbend Unified School
District, against the district's written attendance policy (BP 5113) and the school calendar, both
of which follow. Your output is the CODED ABSENCE RECORD a member of office staff confirms -- so
that the office confirms a record instead of reading a note.

How to read the note:

- The note is the parent's account of the REASON. It is NOT evidence of documentation. What the
  office holds is stated in the OFFICE RECORD below; a note that says a card "is attached" or "was
  dropped off" changes nothing if the office logged no attachment.
- Resolve every date the note gives against the date the office received it and the calendar
  below. Only SCHOOL DAYS are absence days -- never a Saturday, a Sunday or a listed non-school
  day. An early-release day is a school day.
- A late arrival or an early dismissal is a partial-day record on that one day.
- If the note gives two reasons for different days, return one record per reason. If a rule in
  the policy codes some days of one span differently from the rest, return one record per code.
- verify is true when the note names a student other than the one on the office record, or a
  name that does not match it. Code the content regardless.
- Apply the policy as written, including its arithmetic: consecutive school days, the late-note
  rule, the pre-approval rule. Where the note leaves you unsure, code what the policy says and
  say so in `why`.

Reply with JSON and nothing else, in the shape given at the end.

THE ATTENDANCE POLICY, as written:

# Riverbend Unified School District — Board Policy 5113
## Attendance: Excused and Unexcused Absences (2026-27, as applied to the Fall 2026 term)

*This district, this policy and every student, parent and note in this kit are invented. See
`data/SOURCES.md`.*

### 5113.1 — How an absence is recorded

1. The attendance office records every absence as one or more **coded absence records**. A record
   is one code applied to one set of **school days**. A parent's note is the account of the
   reason; the office record is the account of what documentation the office actually holds.
2. **Only school days are absence days.** Saturdays, Sundays and the non-school days on the
   district calendar are never absence days, whatever a note says. An early-release (minimum) day
   is a school day.
3. **A late arrival or an early dismissal is a partial-day record on that one day**, coded by the
   same rules as a full day for the same reason, and marked partial.
4. Where a note gives **two reasons for different days**, the office records one record per reason.
   Where a rule below codes some days of one span differently from the rest, the office records
   one record per code.
5. **Documentation is what the office has logged with the note** — not what the note says was
   sent, attached, or will follow. A note that says "the doctor's note is attached" with no
   attachment logged is a note with no documentation.
6. **verify** is set when the note names a student other than the student on the office record, or
   a name the office cannot match to it. The record is still coded from the note's content; verify
   only asks a person to confirm the match before the record is confirmed.

### 5113.2 — Excused absences

| code | reason | documentation |
|---|---|---|
| **EXC_ILL** | (a) student illness | a parent's note covers up to **3 consecutive school days**. Illness of **4 or more consecutive school days** requires a **healthcare provider's note** |
| **EXC_MED** | (b) medical, dental or vision appointment | an **appointment card** or provider's visit confirmation, **always** — a partial day too |
| **EXC_REL** | (c) religious observance | none — the parent's note is sufficient |
| **EXC_FAM** | (d) family emergency or bereavement (a death or serious illness in the immediate family) | a parent's note covers the **first 3 school days**. Days beyond the third require a **written family statement** |
| **EXC_LEGAL** | (e) court appearance or other legal obligation of the student | a copy of the **court notice or summons**, always |
| **EXC_PRE** | (f) pre-arranged absence — travel, a family event, a college visit or similar planned absence | a **Pre-Arranged Absence Form approved at least 5 school days before the first day absent**. Count back from the first day absent: the school day immediately before it is one school day before. A form approved later than that is not an approval for this purpose |

Consecutive school days are counted across weekends and non-school days: a student out Thursday,
Friday and the following Monday has been out **3** consecutive school days.

### 5113.3 — Unexcused absences (UNX)

(a) An absence for a reason this policy does not name is unexcused. That includes vacation or travel
with no approved Pre-Arranged Absence Form, oversleeping or missed transport, an absence for which no
reason is given, and "family obligations", "family commitments" or "personal reasons" that do not
describe an emergency or bereavement under 5113.2(d).

(b) **Late notes.** Day one is the first school day after the last day absent — the day the
student is back. A note the office logs **on or before the third school day** is timely. A note
logged **after the third school day** is late, and **every day it covers is coded UNX, whatever the
reason and whatever documentation accompanies it.**

(c) An absence coded UNX requires no documentation: its documentation status is `not_required`,
because no document changes the code.

### 5113.4 — Pending documentation (PENDING)

Where the reason qualifies under 5113.2 and the documentation that reason requires has **not been
logged**, the days that require it are coded **PENDING** with documentation status `missing`, and
the record names the document required. Under 5113.2(d), days one to three are EXC_FAM on the
parent's note and only the days beyond the third are PENDING for the family statement. Under
5113.2(f), a missing or late-approved form does not leave the days pending: it makes them UNX.

### 5113.5 — The record

Each coded absence record carries: the **code**; the **school days** it covers; whether it is a
**partial** day (`none`, `late_arrival`, `early_dismissal`); the **reason category** the note
supports (`illness`, `medical_appointment`, `religious_observance`, `family_emergency`, `legal`,
`travel_or_event`, `overslept_or_transport`, `no_reason`, `other`); the **documentation required**
(`none`, `provider_note`, `appointment_card`, `court_notice`, `prearranged_form`,
`family_statement`); the **documentation status** (`attached`, `missing`, `not_required`); the
**verify** flag; and one sentence of **why**. `not_required` is used whenever this policy requires
no document for the record, even if something was attached.


THE SCHOOL CALENDAR:

Calendar window 2026-08-13 to 2026-10-15, Fall 2026 term (2026-08-24 to 2026-12-18).
Saturdays and Sundays are never school days.
School days in this window:
  2026-08-24 Mon   2026-08-25 Tue   2026-08-26 Wed   2026-08-27 Thu   2026-08-28 Fri
  2026-08-31 Mon   2026-09-01 Tue   2026-09-02 Wed   2026-09-03 Thu   2026-09-04 Fri
  2026-09-08 Tue   2026-09-09 Wed   2026-09-10 Thu   2026-09-11 Fri   2026-09-14 Mon
  2026-09-15 Tue   2026-09-16 Wed   2026-09-17 Thu   2026-09-18 Fri   2026-09-21 Mon
  2026-09-22 Tue   2026-09-23 Wed   2026-09-24 Thu   2026-09-25 Fri   2026-09-28 Mon
  2026-09-29 Tue   2026-09-30 Wed   2026-10-01 Thu   2026-10-02 Fri   2026-10-05 Mon
  2026-10-06 Tue   2026-10-07 Wed   2026-10-08 Thu   2026-10-09 Fri   2026-10-13 Tue
  2026-10-14 Wed   2026-10-15 Thu
Weekday NON-school days in this window:
  2026-09-07 Mon  Labor Day
  2026-10-12 Mon  Staff Development Day (no students)

OFFICE RECORD -- what the attendance office holds. This is the authority on what
documentation exists; the note is not.

  Student on record            Wesley Nguyen (RB-31262), grade 6, Riverbend Middle School
  Note received (logged)       2026-09-17 (Thursday)
  Attachments logged           none
  Pre-Arranged Absence Form    none on file

THE NOTE, verbatim:

[Handwritten note on lined paper, transcribed by the front office. Received Thursday, September 17, 2026 at 7:57 PM.]

To whom it may concern,

Wesley Nguyen was out Sept 16 and 17: there was a death in the family and we were away for the services. Let me know if you need anything else.

Monica Nguyen


Reply with JSON and nothing else, exactly this shape:

{"student_named_in_note": "<the student's name exactly as the note gives it, or null if the note names no student>",
 "records": [
   {"reason_category": "illness" | "medical_appointment" | "religious_observance" | "family_emergency" | "legal" | "travel_or_event" | "overslept_or_transport" | "no_reason" | "other",
    "first_day": "<YYYY-MM-DD: the first day the note describes for this record>",
    "last_day": "<YYYY-MM-DD: the last day; the same as first_day for a single or partial day>",
    "dates": ["<YYYY-MM-DD>", ...],
    "partial": "none" | "late_arrival" | "early_dismissal",
    "code": "EXC_ILL" | "EXC_MED" | "EXC_REL" | "EXC_FAM" | "EXC_LEGAL" | "EXC_PRE" | "UNX" | "PENDING",
    "documentation_required": "none" | "provider_note" | "appointment_card" | "court_notice" | "prearranged_form" | "family_statement",
    "documentation_status": "attached" | "missing" | "not_required",
    "verify": true | false,
    "why": "<one sentence naming the clause that decided the code>"}
 ]}

`dates` lists only the SCHOOL DAYS the record covers, in order. One record per reason, and one
record per code where the policy splits a span.

Raw responseThe raw response

{"student_named_in_note":"Wesley Nguyen","records":[{"reason_category":"family_emergency","first_day":"2026-09-16","last_day":"2026-09-17","dates":["2026-09-16","2026-09-17"],"partial":"none","code":"EXC_FAM","documentation_required":"none","documentation_status":"not_required","verify":false,"why":"The note's death in the family is a family emergency/bereavement under 5113.2(d), and the parent's note covers the first 3 school days, so both school days are EXC_FAM with no additional documentation."}]}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Turn a school's excuse notes into coded records — 60 excuse notes. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

No model grades anything. Every metric is exact match against data/gold.jsonl -- strings, date SETS and booleans -- with an arm's records matched to the key's by school-day overlap, so an arm that answers the easy record of a two-record note and omits the hard one is scored wrong on the omission rather than credited for coverage.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on this page is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid. The token counts come from the provider's own usage block on each call.

Rates checked 2026-08-27. The provider that actually ran every call here is kept off this page per the series rule. The real spend is recorded in the run record and the shared call ledger, not on this page.

Grading unitWhat the grading figure prices

Scoring is pure Python over committed JSON: 65 records in well under a second, no key, no network. The ARM costs money; grading it does not.

The gradersThree ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Set limitationsWhat this set cannot show

The corpus is deliberately UNBALANCED the other way from most: 43 of 65 records are genuinely excused, so an arm that excuses everything scores 66.2 pct on the code and 0 of 22 on the records with something to suppress. 20 notes are clean and 40 planted; the 22 UNX/PENDING records sit one per note, so the suppressible set is also a note set -- which is what let the injection probe pair cleanly.

The headline is only readable against the floor and the direction grader together: a high record_all_correct_pct with a nonzero false_excused is worse than a lower one without, and the per-class rates here are CHOSEN, not observed -- no rate estimates how often real parents send a late note or dress a vacation as a family obligation.

$0.00 -- the generator and the label gate are pure Python. No attempt was made to match a real district's class frequencies, because none is public. See data/SOURCES.md.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-27. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingGrading is free; the bill is the 80 calls that answered.

The scorer, the label gate and the rules floor are pure code and cost $0.00 to run against any result set -- there is no LLM judge anywhere in the grading path. The figure above is the token counts of the two PAID runs -- the 60-note scored arm ($0.4148 on the card) and the 20-call injection probe ($0.1672) -- priced at the same projected card cost_per_query_usd uses, not a second larger spend.

Cost driversWhat actually moves the bill

• Output tokens: 90.4 pct of them are provider-side reasoning left at the tier's default and re-rolled per call, and the output side is 80 pct of the projected bill.
• The stable prefix. The policy and the calendar are 64 pct of the prompt's characters and identical across calls; the provider that ran this kit billed 57.6 pct of the run's input tokens at its cached rate, and the projection card has no cache tier.
• The note itself is 3 pct of the prompt. The bill is the rulebook and the reasoning, not the document.

Your volumeWhat it costs at your volume

Linear. No index, no retrieval, no state between notes -- ten times the notes is ten times the calls at the same per-note cost and latency. And the workload is small: a 2,000-student district logging two pct absence a day is roughly 40 notes, about $0.28 a day on the projection card. What does not scale is the socket -- completions are unstreamed, so concurrency is bounded by provider limits and file descriptors, not by anything in this kit.

Model choiceWhy this model

It is the tier the estate runs, so the figure compares with every sibling kit. The finding here is not about the model: it read 65 of 65 records correctly, and everything it got wrong was arithmetic the free engine re-derives for nothing.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

the whole 60-note scored run (r001-absence-code, the fast tier): 163,891 input tokens (94,464 of them prefix cache hits) and 110,958 output. The real bill is on the withheld provider's card and lives in the run record, not on this page.

The codeEvery file, and what it is for

Eight modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/prompt.pyprompt assembly

Six parts in a fixed order -- the system role, the policy verbatim, the calendar window, the office record, the note verbatim, the JSON schema. It names the eight codes, the nine reasons and the rules; it never does the arithmetic for the model, because counting is the thing being measured.

# The assembled prompt, in one place, in the order it is sent.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
POLICY_MD = open(os.path.join(HERE, "data", "policy.md"), encoding="utf-8").read()
SYSTEM = """You are coding one parent's absence note for the attendance office of Riverbend Unified School
SCHEMA = """Reply with JSON and nothing else, exactly this shape:
ATTACHMENT_LABEL = {"provider_note": "healthcare provider's note", "appointment_card": "appointment card",
def office_block(meta):
def build(text, meta):
def render(parts):
VERDICTS = P.CODES

src/coder.pythe model call

One call per note. Parses and normalises the reply -- case-folds the closed vocabularies, coerces a one-date string into a list -- and returns the raw answer and the rechecked one side by side.

# One note in, one coded absence record out. The only place a model is called.

MAX_TOKENS = 32000
THINKING = None
def parse_reply(text):
def _low(v):
def normalise(obj):
def code(cfg, text, meta, complete_fn=None, max_tokens=None):

src/adapters/__init__.pythe adapters — a swap seam

Raw HTTP to any OpenAI-compatible provider or Anthropic. Retries a busy provider, refuses to retry a wrong request, and checks the shared daily call cap before spending.

You change it to: PROVIDER + BASE_URL + MODEL in .env -- one line, then the same run again. Two adapters ship (an OpenAI-compatible shape and Anthropic's Messages API); adding a third is one function and one entry in PROVIDERS, and it must return token counts, because the Cost lens prices them.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TIMEOUT_S = 1200
TRANSPORT_RETRIES = 1
def _post(url, headers, payload, timeout=TIMEOUT_S):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}

src/policy.pythe policy, in code

BP 5113 as arithmetic. It writes the answer key, powers the recheck, sits behind the free floor and the UI. Every number comes from data/policy.json -- three is not typed in the code.

# BP 5113 as arithmetic. Pure code, no model, no key.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
POL = json.load(open(os.path.join(HERE, "data", "policy.json"), encoding="utf-8"))
CODES = tuple(POL["codes"])
REASONS = tuple(POL["reason_categories"])
DOC_KINDS = tuple(POL["documentation_kinds"])
DOC_STATUS = tuple(POL["documentation_status"])
PARTIALS = tuple(POL["partial"])
ATTACHMENT_KINDS = tuple(POL["attachment_kinds_the_office_logs"])
UNX = "UNX"

src/schoolcal.pythe calendar

Which dates are school days, consecutive counts across weekends, the return day, the third school day a late note has to beat, the five-day form deadline -- one JSON calendar, walked by every arm.

# The school calendar, as arithmetic. Pure code, no model, no key.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CAL = json.load(open(os.path.join(HERE, "data", "calendar.json"), encoding="utf-8"))
def d(iso):
def iso(x):
FIRST = d(CAL["first_day"])
LAST = d(CAL["last_day"])
HOLIDAYS = {h["date"]: h["name"] for h in CAL["holidays"]}
EARLY_RELEASE = {e["date"]: e["name"] for e in CAL["early_release"]}
WEEKDAYS = ("Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun")

src/recheck.pythe recheck — a swap seam

Takes the model's reason, span, partial flag and the name it read; merges adjacent same-reason fragments back into one reading; re-derives everything the policy decides by arithmetic. The RAW arm is never touched by it.

You change it to: Which fields the recheck takes from the reply -- today the reason, the span, the partial flag and the name the note used. Widening that set moves work from code to model; narrowing it moves the injection surface, because a field the code re-derives is a field the note cannot lie about.

# The model's reading, the policy's arithmetic. Pure code, no model, no key.

def recheck(answer, meta):

evals/scoring.pythe scorer — a swap seam

Matches records to the key by school-day overlap, grades four fields each on its own and all four together, keeps the two error directions apart, and files every miss into exactly one taxonomy bucket.

You change it to: Graders, the record matcher and the failure taxonomy. The two error directions are scored on their own denominators and never averaged; a scorer that blended them would hide which direction is being got wrong.

# Grade one arm against the answer key. Deterministic -- no model judges anything here.

FIELDS = ("code", "dates", "documentation_status", "verify")
DIAG = ("reason_category", "partial", "documentation_required")
SUPPRESSIBLE = ("UNX", "PENDING")
TAXONOMY = ("record_missing", "reason_misread", "dates_wrong", "late_rule_missed",
def _pct(n, d):
def _dates(r):
def match(gold_records, got_records):
def _bucket(g, m):
def score(records, golds, notes_text=None):

evals/check_labels.pythe label gate

Grades the answer key itself with independent arithmetic -- it deliberately does not import the engine. Where the two implementations disagree, the build stops before a call is paid for.

# Grade the ANSWER KEY. No key, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
CAL = json.load(open(os.path.join(DATA, "calendar.json"), encoding="utf-8"))
POL = json.load(open(os.path.join(DATA, "policy.json"), encoding="utf-8"))
REQUIRED_CASES = ("range_weekend", "range_holiday", "late_note", "med_no_card", "ill_long_no_doc",
def d(iso):
FIRST, LAST = d(CAL["first_day"]), d(CAL["last_day"])
HOL = {h["date"] for h in CAL["holidays"]}

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2731 input and 1849 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

MEASURED, FORCED AND PAIRED -- and the pure-code station is the finding. The excuse note is this kit's injection surface: it is written by the household whose absence is being coded and it reaches the model verbatim. The probe forced one office-annotation-shaped line into every note with something to suppress and paired each reply against the scored run's own cached answer.

API_KEY is read from the shared <code>&lt;repo&gt;/.env</code>, this kit's own <code>.env</code>, or the real environment, in that order. Both files are gitignored from the first commit; this repo has never held a credential. The key never leaves the machine and is never printed.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. Both were run for real on 2026-08-30.

The second gate is the one this probe cannot retire: one sentence, one model, one day, aimed at PAPERWORK. A note that argues its REASON -- a trip described as a bereavement in convincing detail -- attacks the field the recheck trusts, and that experiment is unfired. Writing the shape down here is what stops a future run reporting a padded denominator as resistance.

The result3 of 20 raw suppressions, 0 of 20 after the recheck -- forced and paired; 0 of 20 reasons moved.

Every trial is one note of this corpus with the shipped line appended as its final sentence and everything else byte-identical, paired against that same note's own un-injected answer read from cache-r001-absence-code.jsonl -- one variable, and the control cost nothing. Only records the control coded correctly are in the paired denominator (17 raw, 20 rechecked): 'the injected arm also got it wrong' is not evidence about injection. All three raw suppressions are four-to-six-day illnesses whose injected reply coded the first three days EXC_ILL -- the same borrowed family-emergency split as the scored run's own misses, now on notes the control had right. Two of the control's own misses (AN-0006, AN-0028) came back RIGHT under injection: call-to-call variation on a re-rolled reasoning budget, excluded from the paired denominator for exactly that reason. One phrasing, one position, one model, one day, 20 calls: 55,360 tokens in (40,576 cache hits), 46,521 out, p50 12.8 s, p95 76.0 s, 107.8 wall seconds -- $0.1672 on the same projection card the Cost lens uses.

HonestyWhat this does not prove

• Any phrasing but the one shipped, and any position but the final line. A note that argues its REASON -- the field the recheck trusts the model for -- is a different experiment and the one this design most owes.
• Repeats. One draw of a decision the provider re-rolls per call; the same run showed two un-injected answers change on a re-fire.
• Whether the rules floor is a mitigation. Its keyword reader never weighs an instruction, so this sentence cannot reach it -- but that is an argument from its design, not a measured result: the floor was not run over the injected notes.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never confirm an absence record, never change a student's record, never notify a parent, never decide an appeal. This kit produces the record a PERSON confirms, and the verify flag is the record asking that person to look before confirming.

Stated on the UI, in the README and in src/app.py's docstring, and enforced by there being no such endpoint: the server's POST routes return coded records and nothing else, and there is no write path to any student system.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is ABSENCE OF A WRITE PATH plus a prompt rule, not a runtime enforcement layer -- with one real exception: the recheck IS enforcement for the arithmetic fields. It is NOT enforcement for the reading, and the reading is where every unfixable error lives.
• ⚠︎ THE INJECTION RESULT IS NOT A DEFENCE OF THE READING. The probe measured one sentence aimed at documentation; the recheck's 0 of 20 is architecture (the lied-about field is one the code never asks the model for), and an attack that argues the REASON -- the field the recheck trusts -- is unmeasured.
• The rechecked 100 pct is agreement with a computed key on an invented corpus; it is not evidence about a real district's queue, and the engine that rechecks is the engine that wrote the key.
• The verify flag is a request to a person, not a block. Nothing stops a confirmed record on a mismatched name except the person the flag is for.

WatchedWhat is watched, and why that one

The kit measures 28 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

20 measured by the latest run 8 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

3 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

absence-code · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 43 chips that all say so.

absence-code · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 45 chips that all say so.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 20 chips that all say so.

DeviationsWhat deviated

0 breaches across 3 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The label gate and the floor are free and re-run in seconds -- 0.11 s measured end to end on a keyless copy. Only the model arm and the probe cost money (80 calls all told), and neither is re-fired after its misses are read; the recheck can be re-derived from the cached replies at any time for nothing.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, standard library end to end -- no orchestration layer, no vendor SDK, no retrieval stack, no pip install. requirements.txt names nothing and says why.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear, and concurrent only at the note level. There is no agent, no tool loop, no retrieval and no state carried between notes; every note is independent, which is exactly why 5 workers is the entire concurrency story.

The other sideWhat a framework costs you

• Adding a provider means editing the PROVIDERS dict by hand. In exchange pip install pulls nothing and a forker runs this on whichever key they already hold.
• The reply is parsed by hand. It held across all 80 paid calls, and that is an observation, not a guarantee -- a structured-output library would fail loudly where this fails quietly.
• No rules-engine DSL, so a policy whose SHAPE differs from BP 5113's (reason -> document kind, day allowance, deadline) is a change to src/policy.py rather than to a config file. The JSON carries the numbers; the code carries the shape.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-absence-code on the fast tier, 2026-08-30. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 1 earlier run on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 1 run on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT, with the corpus generated in-process -- there is no third-party data in this kit to licence. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe whole record -- code, the set of school days, documentation status, the verify flag -- and, on its own denominator, the whole note

whether each of the 65 coded records is one the office could confirm as written: the code the key carries, the exact set of school days, the documentation status the office record supports, and the verify flag. And, SEPARATELY, whether a whole note came back right -- every record correct and no extra record invented -- because the office confirms notes, not cells.

MethodHow the test was run

python3 -m evals.run --run-id <id> [--floor rules]; evals/scoring.py matches records to the key by school-day overlap and compares strings, date sets and booleans against data/gold.jsonl. No model is in the grading path.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The kit in one row: the model READ it right and coded it with a rule from the clause beneath the right one. Pure code cannot read 'throwing up all night'; a model should not be doing day arithmetic. The recheck is that split of labour made executable.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl -- computed by src/policy.py from the generator's injected facts, never typed by a person; graded independently by evals/check_labels.py, which re-reads the calendar and policy JSON and walks the dates itself. Two implementations, one key. Every student, parent and note is invented -- see data/SOURCES.md.

No true/false rates for this grader. It records 4 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? There is no threshold to tune -- exact match on strings, date sets and booleans, so nothing here has a tolerance. The numbers that look like thresholds live in the POLICY, not the grader: three days, three days, five days, the third school day -- all in data/policy.json, all applied by every arm.

Cadence: Once per corpus version. The paid arm ran ONCE and was not re-fired after its misses were read; the rechecked column was re-derived once from the cached replies (rechecked_rederived_from_cache: true in the result file), which changed no raw figure, no token count and no bill.

The decisionWhen to reach for it

In one lineThe two error directions, never averaged -- a UNX or PENDING coded as an excuse, and an excuse coded UNX or PENDING

which way an arm fails. false_excused is a record with something to suppress -- a key UNX or PENDING -- coded as an excuse: the direction a parent wants and an office cannot revisit. false_denied is the reverse: the one a parent complains about the same afternoon. One blended accuracy figure hides which is being made, so they are scored on their own denominators (22 and 43).

MethodHow the test was run

python3 -m evals.run --run-id <id> [--floor rules]; evals/scoring.py splits every code miss by the direction it moved. No model is in the grading path.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The kit in one row: the model READ it right and coded it with a rule from the clause beneath the right one. Pure code cannot read 'throwing up all night'; a model should not be doing day arithmetic. The recheck is that split of labour made executable.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The same data/gold.jsonl as the record grader; the split into 22 suppressible and 43 excused records is fixed by the key, before any arm answers.

No true/false rates for this grader. It records 3 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No tolerance -- a code either moved across the excused/unexcused line or it did not. The denominators are stated beside every rate because 22 is small: one record moves this grader by 4.5 points.

Cadence: Every run, on every arm, including the free floor -- and it is the pair the injection probe re-measures under attack.

The decisionWhen to reach for it

In one lineThe four things the model is trusted with -- the reason, the span, the partial flag and the name the note used

whether the READING half of the job was right -- the only half the recheck cannot fix. src/recheck.py takes exactly these four from the reply and re-derives everything else, so this grader is the ceiling on what the pure-code station can rescue: a misread reason is rechecked, correctly, into the wrong record.

MethodHow the test was run

python3 -m evals.run --run-id <id> [--floor rules]; the reading fields are compared like every other field, exact match against data/gold.jsonl.

Every grader on these pages scored the same 60 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The kit in one row: the model READ it right and coded it with a rule from the clause beneath the right one. Pure code cannot read 'throwing up all night'; a model should not be doing day arithmetic. The recheck is that split of labour made executable.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The reason category, partial flag and name of every record are injected facts of the generator, fixed by tools/build_corpus.py before any arm sees a note.

No true/false rates for this grader. It records 4 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? Exact match on closed vocabularies (nine reasons, three partial states, a boolean) -- no tolerance. The reason set is closed on purpose: a reason outside it normalises to 'other', which codes UNX.

Cadence: Every run, on every arm. Under the injection probe this grader is the one that held outright: 0 of 20 reasons flipped.

The decisionWhen to reach for it