The business caseThe problem this solves

Deciding whether a student account's assessed tuition and fees are right is arithmetic with an order, and every step of it has a trap: a credit threshold that is inclusive and where the flat band is cheaper than the per-credit rate, a differential fee that applies to some course levels and not others, a mandatory fee most waivers do not touch, and a residency reclassification that changes the next term rather than this one. Next to all of it sits the bursar's own free-text note, which is the loudest thing on the record and the one part of it that is nobody's measurement. Someone opening each student account, reading the residency tier of record, the enrolled credit load, the course level and any waiver, running four steps of the rate table in the right order — credit band, then the course-level differential fee, then the mandatory fee, then the waiver, which is applied last and does not cover everything — and comparing the answer against what the bursar system actually assessed. It is a couple of minutes per account and it is every account in the term, and the part that goes wrong is rarely the addition: it is treating twelve credits as twelve times the per-credit rate, re-pricing the term because a residency reclassification is sitting on the account, or letting the bursar's own note set the answer.

The inputThe actual account assessments

The corpus is 55 account assessments, 0.03 MB (txt 55). Plain text, one format, invented rather than fetched — tuition assessment is an education-records domain, and a real record names a real student, their enrolment, their aid and often their residency history. There is no public corpus of (assessment record, correct total) pairs for the same reason there is no public corpus of bank statements. Generating it also makes the label mechanical: gold is the rate table run over the record's own values, not somebody's reading of the bursar's note — and it lets the generator assert that every planted variance has exactly one single-rule explanation, which is what makes the reason field scoreable at all.

Student Account
---------------
SA-WM-87980

Campus
------
Fairhaven Campus

Term
----
2026-SU

Residency Tier
--------------
In-State

Enrolled Credits
----------------
16 credit hours

Course Level
------------
Lower Division

Waiver
------
Staff Dependent Waiver

Assessed Total
--------------
3520 USD

Bill Status
-----------
posted

Residency Action
----------------
None on file for this term.

Bursar Notes
------------
Student disputed their residency classification; account under manual audit.

The outcomeWhat a good result looks like

A twelve-field extracted record per account — including two decided fields: whether the assessed total matches the rate table, and which single step of the table it departs from — plus one pure-code routing decision taken from two of those fields: a mis-assessed account whose bill has already posted is the one that goes to the bursar today. Nothing here re-bills an account, reverses a charge, adjusts aid or contacts a student.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Screening a term's assessments for accounts charged the wrong amount, before a bursar analyst opens them — either tier — both answered every account they saw correctly, 1.00 precision and 0.963 recall measured here
  98.18 pct verdict accuracy on both tiers against the free bursar-note floor's 60.0 pct (22 wrong, including 9 mis-assessed accounts called correct, the largest a $4,300 variance).
• Needing the VARIANCE REASON, not just the yes/no — routing a correction to the right desk, or reporting on which rule breaks most often — the deliberating tier
  It is the only published figure that separates them: 96.3 pct on the reason against the fast tier's 92.59 pct, and it named every reason it answered correctly. It also returned 648 of 648 cells exact against the fast tier's 647.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-tuition-assess. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts, the 55-record picker populates and the field table draws its twelve empty rows; clicking Extract returns the no-key sentence rather than a stack trace. python -m evals.check_labels passes with no network access at all, and python -m evals.run --baseline scores the free floor over all 55 records for free.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per student account assessment

Current processWhat it replaces

Someone opening each student account, reading the residency tier of record, the enrolled credit load, the course level and any waiver, running four steps of the rate table in the right order — credit band, then the course-level differential fee, then the mandatory fee, then the waiver, which is applied last and does not cover everything — and comparing the answer against what the bursar system actually assessed. It is a couple of minutes per account and it is every account in the term, and the part that goes wrong is rarely the addition: it is treating twelve credits as twelve times the per-credit rate, re-pricing the term because a residency reclassification is sitting on the account, or letting the bursar's own note set the answer.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings; a record with none falls back to one whole-document segment

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 55 records cut into 605 sections in under a hundredth of a second, in process, with no model and no network.

LicenceLicence

MIT — this repository's own licence. Every account id, campus name, term code, credit load, waiver, dollar figure and bursar note is invented; NO REAL STUDENT DATA of any kind is present, and no real institution, published tuition schedule, fee bill, waiver programme or residency regulation is named or reproduced.

Bring your ownBring your own account assessments

Replace data/corpus/*.txt, write your own data/fields.json, and supply a gold record per account. SECTION_HINTS in src/select.py maps fields to section headings and will need editing for a different bursar extract; when it does not match, selection falls back to the whole document — slower, more expensive, always correct. assess() and compute() in src/extract.py are this kit's own invented rate table and routing rule and should be the first things you replace with your own board-approved schedule.

⚠︎ And what stops being true when you do: Every score on this page was measured on a corpus where each account departs from the rate table in AT MOST ONE place and the variance vocabulary has exactly five members. Point it at your own accounts and that stops being true first: a real mis-assessment with two causes has no single correct variance_reason, so the reason grade does not carry over even when the yes/no verdict does.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 1,963 tokens assembled, 934 are instructions — 48% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on STU-0025 — three calls at max_tokens=1, each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. The three parts sum to 1963, which is exactly the input_tokens the provider billed for the example call in results/example-STU-0025.json. kits/UC0041-tuition-assess/results/tokens-p001-tuition-assess.json.

Full promptThe prompt in full

You extract structured fields from a student-account tuition assessment record for one student, for one academic term. You return JSON and nothing else.

RULES, in order of importance:
1. If the record does not state a field, return null for it. Do not infer it and do not use what you know about the world.
2. `assessment_correct` is decided by ARITHMETIC over four structured values -- residency_tier, enrolled_credits, course_level and waiver_type -- compared against the assessed_total_usd the record already states. It is never decided by how the bursar's note reads and never by the residency action line. Compute the correct total yourself, in this order:
   a. TUITION. enrolled_credits of 12 or more is a FULL-TIME load, and tuition is the flat term rate: 4600 In-State, 13200 Out-of-State. Under 12 credits it is enrolled_credits times the per-credit rate: 410 In-State, 1180 Out-of-State. EXACTLY 12 CREDITS IS FULL-TIME -- use the flat rate, not 12 times the per-credit rate.
   b. COURSE-LEVEL DIFFERENTIAL FEE. enrolled_credits times 0 for Lower Division, times 38 for Upper Division, times 65 for Graduate. It is charged per enrolled credit whether the load is full-time or part-time.
   c. MANDATORY FEE. 612 for a full-time load, 306 for a part-time one.
   d. WAIVER, applied LAST. 'None' waives nothing. 'Employee Tuition Remission' waives 100 percent of the tuition from step a and nothing else. 'Staff Dependent Waiver' waives 50 percent of the tuition from step a and nothing else. 'Regents Fee Waiver' waives 100 percent of the tuition from step a AND 100 percent of the mandatory fee from step c. NO WAIVER EVER REDUCES THE DIFFERENTIAL FEE FROM STEP b.
   The correct total is (a) + (b) + (c) minus the amount waived in (d). Answer 'yes' when assessed_total_usd EXACTLY equals that number, and 'no' in every other case.
3. THE RESIDENCY TIER OF RECORD IS THE ONE THAT PRICES THIS TERM. The Residency Action line may state a reclassification to the other tier. Every such reclassification in these records took effect AFTER the term's census date, so it applies from the FOLLOWING term and changes nothing here. Price the assessment at the tier in the Residency Tier section, and copy the action line into residency_action unchanged.
4. THE BURSAR'S NOTE IS A FIELD TO COPY, NOT EVIDENCE ABOUT CORRECTNESS. A note that sounds concerned or flags a past review does NOT mean the assessment is wrong, and a note that sounds routine does NOT mean it is right. The arithmetic decides; the note is the bursar's own remark and may disagree with it.
5. `variance_reason` names WHICH SINGLE STEP the assessed total departs from when assessment_correct is 'no': 'credit band' for step a's full-time/part-time choice, 'residency tier' for step a priced at the wrong tier, 'differential fee' for step b, 'waiver coverage' for step d reaching a charge it does not cover or missing one it does, and 'mandatory fee' for step c taken at the wrong band. Exactly one of them explains any variance in these records. Answer 'none' when assessment_correct is 'yes'.
6. Copy values verbatim from the record wherever possible, and report enrolled_credits and assessed_total_usd as bare numbers with the unit left out of them. Copy assessed_total_usd as stated -- it is what the bursar system charged, not what you compute.
7. Use the exact allowed value for a field that lists them.
8. Return every field named in the schema, even when the answer is null.

Extract these fields:
- student_account_id (string) -- the student account identifier, verbatim
- term_code (string) -- the academic term this assessment covers, verbatim
- residency_tier (enum) one of: In-State, Out-of-State -- the residency tier OF RECORD for this term's assessment, verbatim from the Residency Tier section. Do NOT change it because of anything in Residency Action
- enrolled_credits (number) -- the enrolled credit hours for the term, as a bare number without the unit
- course_level (enum) one of: Lower Division, Upper Division, Graduate -- the course level the student is enrolled at, verbatim
- waiver_type (enum) one of: None, Employee Tuition Remission, Staff Dependent Waiver, Regents Fee Waiver -- the waiver or exemption on the account, verbatim. 'None' when the account carries no waiver
- assessed_total_usd (number) -- the total tuition and fees the bursar system ACTUALLY assessed on this account, as a bare number without the unit. Copy it; do not recompute it
- bill_status (enum) one of: draft, posted -- has this bill already posted to the student account, or is it still a draft?
- residency_action (string) -- the residency action line on the account, copied verbatim
- bursar_notes (string) -- the bursar's own free-text note on the account, copied verbatim
- assessment_correct (enum) one of: yes, no -- does assessed_total_usd match what the rate table entitles this account to? Decide this STRICTLY by arithmetic over residency_tier, enrolled_credits, course_level and waiver_type -- never from bursar_notes and never from residency_action. Compute the correct total in this order: (1) TUITION -- enrolled_credits of 12 or more is full-time, so tuition is the flat term rate (In-State 4600, Out-of-State 13200); under 12 it is enrolled_credits times the per-credit rate (In-State 410, Out-of-State 1180). (2) DIFFERENTIAL FEE -- enrolled_credits times 0 for Lower Division, 38 for Upper Division, 65 for Graduate. (3) MANDATORY FEE -- 612 for a full-time load, 306 for a part-time one. (4) WAIVER, applied last -- None waives nothing; Employee Tuition Remission waives 100 percent of tuition only; Staff Dependent Waiver waives 50 percent of tuition only; Regents Fee Waiver waives 100 percent of tuition AND 100 percent of the mandatory fee. NO waiver ever reduces the differential fee. The correct total is tuition plus differential fee plus mandatory fee minus the waived amount. Answer 'yes' when assessed_total_usd EXACTLY equals that number, 'no' in every other case
- variance_reason (enum) one of: none, credit band, residency tier, differential fee, waiver coverage, mandatory fee -- when assessment_correct is 'no', WHICH SINGLE step of the rate table the assessed total departs from -- 'credit band' (the full-time flat rate applied to a part-time load or the per-credit rate applied to a full-time one), 'residency tier' (tuition priced at the tier named in Residency Action instead of the tier of record), 'differential fee' (charged at the wrong course level's per-credit rate, including charged on Lower Division or omitted on Upper Division or Graduate), 'waiver coverage' (the waiver reached the mandatory fee when its type does not cover it, or failed to when its type does), 'mandatory fee' (the per-term fee taken at the wrong full-time/part-time rate). Exactly one of these explains any variance in this corpus. Answer 'none' when assessment_correct is 'yes'

Return a JSON object with exactly these keys: student_account_id, term_code, residency_tier, enrolled_credits, course_level, waiver_type, assessed_total_usd, bill_status, residency_action, bursar_notes, assessment_correct, variance_reason
Use null for any field the record does not state.

STUDENT ACCOUNT TUITION ASSESSMENT
----------------------------------
Student Account
---------------
SA-NJ-79361

Term
----
2026-FA

Residency Tier
--------------
In-State

Enrolled Credits
----------------
12 credit hours

Course Level
------------
Graduate

Waiver
------
None

Assessed Total
--------------
6312 USD

Bill Status
-----------
posted

Residency Action
----------------
Reclassified to Out-of-State effective 2026-10-27, after the term census date of 2026-09-14.

Bursar Notes
------------
Charges look normal for this enrollment pattern, all in order.

Raw responseThe raw response

{
  "student_account_id": "SA-NJ-79361",
  "term_code": "2026-FA",
  "residency_tier": "In-State",
  "enrolled_credits": 12,
  "course_level": "Graduate",
  "waiver_type": "None",
  "assessed_total_usd": 6312,
  "bill_status": "posted",
  "residency_action": "Reclassified to Out-of-State effective 2026-10-27, after the term census date of 2026-09-14.",
  "bursar_notes": "Charges look normal for this enrollment pattern, all in order.",
  "assessment_correct": "no",
  "variance_reason": "credit band"
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check a student's tuition bill against your rate table — 55 account assessments. Two tiers of one model family answered, and every answer was then graded Four different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran r001 and r002 publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersFour ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same assessment, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The bursar-note floor (evals/baseline.py) and the scorer (evals/judge.py) are both pure code and cost $0.00 to run against either result set. The figure above is both runs' own token counts (0.1363 on the fast tier, 0.1547 on the deliberating tier) priced at Google Gemini 3 Flash's published rate — the same basis cost_per_query_usd uses, not a second, larger spend. The calibration run, the prompt-token measurement and the worked-example capture add seven more calls that are not in this figure.

Cost driversWhat actually moves the bill

• The fixed system prompt and field schema (1757 of 1963 tokens on the example call, 90 pct) outweigh the record sections sent (206 tokens) by nearly nine to one — the floor every call pays before a single value is read. This kit's rate table is stated in full in the prompt, which is why its fixed prefix is larger than its siblings'.
• Output length: the model returns a full twelve-key JSON record every call, including the residency action line and the bursar's note copied back verbatim, whatever the record says — 516 output tokens per account on the fast tier and 630 on the deliberating one.

Your volumeWhat it costs at your volume

Linear in accounts: each call is independent, carries the same fixed prompt and shares nothing with its neighbours, so 550 accounts cost ten times 55 and take ten times as long. Nothing amortises — there is no index to build and no cache, which is also why there is no volume discount to find without changing the design. The shape that would change it is prompt caching on the 90 pct fixed prefix, which nothing here does.

Model choiceWhy this model

The fast tier is what r001 ran and what the flow figure and the UI shot are drawn from. The deliberating tier is the only thing that moved a published score — one variance reason and one extraction cell — so the recommendation splits by what you need out of the kit rather than by a single winner.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 55 student account assessments attempted on the fast tier, 54 of which returned a reply. The deliberating tier's own calls are recorded separately in Cost.cost_by_model and are not projected here.

The codeEvery file, and what it is for

Five modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the student account record into addressable sections, pure code

# Cut a document into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections carry each field, pure code — the two decided fields are mapped to the five facts the rate table reads (residency tier, credits, course level, waiver, assessed total) and to NEITHER decoy, and the Campus section is mapped by nothing and never sent

You change it to: map fields to your own bursar extract's headings; unmatched falls back to the whole document

# Pick which sections plausibly carry each field. Pure code -- the last deterministic step before

SECTION_HINTS = {
def for_field(secs, field):
def plan(secs, fields):

src/prompt.pyprompt

assemble one call for all twelve fields, with the whole rate table stated in full — the inclusive full-time threshold, the differential fee that is zero for Lower Division, the mandatory fee, and the waiver applied last and never to the differential

# Assemble the extraction prompt. One prompt per student-account tuition assessment, all twelve

SYSTEM = (
def field_schema(fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key — plus the pure-code business-condition check downstream: a mis-assessed account whose bill has already posted is routed for a bursar correction

You change it to: the routing rule itself — this kit ships two booleans (mis-assessed AND already posted) and a real bursar's office weighs the dollar size of the variance, whether aid has disbursed against it, and any refund-deadline rule. It is one function, and it is deliberately not the same function as assess(), so changing WHO GETS ROUTED does not change WHAT CORRECT MEANS

# Extract one student-account tuition assessment's fields: segment, select, prompt, one model

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 4000
FULL_TIME_CREDITS = 12
PER_CREDIT = {"In-State": 410, "Out-of-State": 1180}
FLAT_TERM = {"In-State": 4600, "Out-of-State": 13200}
DIFFERENTIAL = {"Lower Division": 0, "Upper Division": 38, "Graduate": 65}
MANDATORY_FULL = 612

evals/judge.pyjudge

score field accuracy, the assessment confusion matrix, the variance-reason grade and the review flag separately, pure code

# Score an extraction run. PURE CODE -- gold is exact and the answer is one value per cell, so

def norm(v):
def _number(v):
def equal(field, got, want):
def score(fields, records, golds):
def _matrix(rows, positive):
def score_reasons(records, golds):
def score_flags(records, flags, golds):
def variance_sizes(records, golds):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 1952 input and 515 output tokens per assessment, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's student account records are entirely synthetic (tools/build_corpus.py, seed 20260822): NO REAL STUDENT DATA is present, no real institution or student exists in the corpus, and nothing was fetched from anywhere. That matters more here than on most kits — tuition assessment is an education-records domain, and a real record would carry a named student, their enrolment and their aid. The only outbound traffic the kit makes is one chat-completion request per record to the configured provider, carrying the mapped sections of one account. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit and the kit-local one pins only a model name. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser.

The experimentWe did not attack it — and three of four boundaries hold

The three boundaries that hold were confirmed by reading the code, not by a run: no code path bills anybody or contacts anybody; the routing rule and the rate table read only enums and numbers, so neither free-text field can reach them; and the key and base URL are redacted out of any error before it is returned. The fourth is open. An indirect prompt injection needs a field an outside party authored, and this kit has two of them — the bursar's note and the residency action line — which makes it the obvious place to attack and the reason the gap is named rather than glossed. Confirmed by reading the code, not by a run, on 2026-08-22 -- no attack run was fired; see redteam.why.

The first three boundaries hold, confirmed by reading the code rather than by an attack run. The fourth is open and is named as open — an indirect prompt injection in either free-text field is exactly the attack this kit's shape invites, and nothing here has tried it.

The result0 attack trials, and three of four boundaries checked here hold on evidence. The one that does not is the one this kit's own shape invites twice over: it carries TWO free-text fields an outside party wrote, and whether an instruction hidden in either could move the verdict is unmeasured.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted ambiguity is a REGISTER mismatch in one field and a genuinely-inapplicable fact in the other — a routine-sounding note on a mis-assessed account, and a real reclassification that applies to the next term. Both measure whether a model runs the arithmetic when the prose points elsewhere. Neither measures whether a model obeys an instruction hidden in the same field. Those are different failures and only one of them is measured here.

HonestyWhat this does not prove

• Whether an indirect prompt injection in bursar_notes or residency_action could move assessment_correct on either tier. No attack run exists.
• Whether the kit behaves safely against a hostile provider — a response body crafted to break the JSON extraction in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed record, and nothing beyond that was tested. The one truncated reply in r001 exercised that path for real and was recorded correctly.
• Whether a real bursar extract would carry anything sensitive this kit mishandles. The corpus has no personal data by construction; a real record carries a named student, an enrolment history, aid awards and often a residency file, and none of that path is exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_review fires when the assessed total is wrong AND the bill has already posted — assessment_correct == "no" and bill_status == "posted". Both values come out of the same reply; the rule is run afterwards in pure code, over whatever the model returned, and never over gold. A reply missing either value returns None rather than False: an unknown is not a pass.

src/extract.py::compute(), called by extract() on every record, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by identical code. The arithmetic it reads lives in a SEPARATE function, assess(), which is also what the corpus generator wrote gold with and what src/prompt.py states to the model — one definition, three readers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a check on whether the extracted values are right. If the model misreads the credit load and then judges that misreading consistently, the account is routed (or not routed) on a wrong number and this rule cannot tell. The consistency diagnostic in evals/judge.py is the closest thing to that check, it is reported separately, and it is blind to the same case.
• It is NOT any institution's billing-adjustment policy. Mis-assessed-and-posted is this kit's own simplification, invented for this corpus. No published schedule, regulation or bursar procedure was consulted, and none is reproduced. A real office weighs the dollar size of the variance — which here runs from $110 to $8,600 — whether aid has disbursed against it, and any refund-deadline rule.
• It is NOT a disposition. Nothing here re-bills an account, reverses a charge or adjusts aid, and assessment_correct is an extracted field rather than a decision anybody is bound by.
• It is NOT a completeness check. An account that returned no reply at all is not flagged by this rule and is not flagged by anything else in the kit either — it is simply absent from the scored set. Both tiers lost one that way.

WatchedWhat is watched, and why that one

The kit measures 19 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 12 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

3 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 3 runs measured the same system — they differ on assessments, extraction_cells, failures, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 3 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() or assess() in src/extract.py, on any change to data/fields.json's allowed values for assessment_correct or bill_status, and on any corpus regeneration. Changing WHO GETS ROUTED is a policy change and must be re-scored, even though it never changes what 'correct' means.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per record — segment, select, prompt, one call, parse, compute — with no branch, no loop and no agent. Anything that looks like orchestration in a kit this size is a diagram of a straight line.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain — and this run paid for that: the retry/backoff policy covers HTTP 408/429/5xx and not a socket timeout, which cost the deliberating tier one record out of 55. A framework's HTTP client would probably have retried it.
• No framework means no framework's ecosystem — no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. What ships is what is in the repository.
• The provider abstraction covers exactly two wire formats. A third provider is one function and one dict entry, and until somebody writes it the kit runs on two shapes.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-tuition-assess on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 2 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

1 run not plotted. b000-tuition-assess-rules recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Every account id, campus name, term code, credit load, waiver, dollar figure and bursar note is invented; NO REAL STUDENT DATA of any kind is present, and no real institution, published tuition schedule, fee bill, waiver programme or residency regulation is named or reproduced. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-field exact match, light normalisation

Does the model's value for each field match gold, after trimming whitespace/punctuation and treating numbers within half a cent as equal? No field in this corpus is nullable — every one is stated on every record — so a blank is always a miss and never a legitimate absence.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every call, and the same field-match logic the free baseline is scored by — a baseline and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same residency tier, credit load, course level, waiver and assessed total the document states; evals/check_labels.py asserts every field is populated on every row, that every verdict agrees with its own arithmetic, that reason and verdict never disagree, and that every variance has exactly one single-rule explanation, before any run is allowed to spend.

How tight can the band be? There is no tolerance band on the field grade itself — it is exact match after trimming whitespace/punctuation, and numbers within half a cent are treated as equal, never a continuous score to round.

Cadence: Re-score on any change to src/prompt.py, data/fields.json or tools/build_corpus.py — the first two change what is asked, the third changes what is asked about. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineassessment_correct against gold's own arithmetic

Of every account that really is mis-assessed, how many did the run call mis-assessed — and how many correctly assessed accounts did it wrongly flag? VARIANCE IS THE POSITIVE CLASS: an account charged the wrong amount that gets called correct is the failure a bursar's office actually pays for.

MethodHow the test was run

evals/judge.py::score_flags, in-process, no key and no model.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's correctness is re-derived inside the grader by the same four-step rate table the kit publishes — inclusive credit threshold, differential fee by course level, mandatory fee by band, waiver applied last — so the truth this matrix grades against can never be a separately-typed label that drifted from the table.

How tight can the band be? No threshold — the verdict is one of two allowed values, and a reply that returns neither is counted as unanswered rather than folded into the correct-negative cell.

Cadence: Re-run whenever assess() in src/extract.py changes, whenever the corpus is regenerated, and on any provider or model change.

The decisionWhen to reach for it

In one linevariance_reason on the records that actually carry a variance

When the assessed total is wrong, did the model name the right step of the rate table? Scored over the 27 mis-assessed records ONLY, with the opposite-direction error — naming a reason on an account that is correctly assessed — counted beside it and never inside it.

MethodHow the test was run

evals/judge.py::score_reasons, in-process, no key and no model.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The generator's own fault label, re-checked independently: evals/check_labels.py recomputes all five single-rule departures for every mis-assessed record and fails if more than one of them reproduces the assessed total. A number two mistakes could both explain has no true reason, and would make this grader meaningless.

How tight can the band be? Exact match on a six-value enum, no partial credit. A missing reason counts as wrong, which is why the truncated and timed-out records each cost their tier a point here as well as on the verdict.

Cadence: Re-run whenever REASONS in src/extract.py or FAULTS in tools/build_corpus.py changes — widening the vocabulary changes what a correct answer is.

The decisionWhen to reach for it

In one lineneeds_review against the same rule run over gold

Does the pure-code routing decision — mis-assessed AND already posted — land on the same records it would land on if both fields had been read perfectly? It is a BUSINESS CONDITION, so unlike a self-consistency check it genuinely needs labels, and saying so is half of what makes the number believable.

MethodHow the test was run

evals/judge.py::score_flags, in-process, no key and no model.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute(), the same function the run uses, applied to GOLD's assessment_correct and bill_status. One rule, two inputs, so a change to the rule moves both sides together and the grader cannot silently grade an old policy.

How tight can the band be? No threshold — two booleans and an AND.

Cadence: Re-run whenever compute() changes. Changing WHO GETS ROUTED is a policy change and must be re-scored, even though it never changes what 'correct' means.

The decisionWhen to reach for it