The business caseThe problem this solves

A plan has to carry the components the rules require, and each of those components has to be written in terms somebody can measure against a year later. Those are two different checks and only the first one is easy. A missing section has a missing heading and any checklist finds it. A section that is THERE and cannot be measured against -- a goal with no baseline, no criterion or no measurement method; a service with no frequency; a reporting schedule that says 'regularly' -- passes every checkbox review, gets a tick, and is discovered at the next review when nobody can say whether the goal was met. The plan also puts two things next to the answer that look like evidence and are not: a previous reviewer's own tick-box result, printed on the plan, and the case manager's free-text remark before the meeting. Somebody opening each education plan, holding a list of the components the rules where they work require, and reading the plan section by section against it. The part that is quick is ticking off the headings. The part that takes the time, and the part that gets skipped when the meeting is tomorrow, is reading INSIDE each section to see whether what is written there can actually be measured against at the next review -- whether every goal states a baseline, a criterion and a method, whether every service line states its frequency, duration and location, whether the reporting schedule is a schedule rather than the word 'regularly'. And there is one component whose requirement depends on the pupil rather than the plan, so a reviewer who ticks the same seven boxes on every plan raises a false defect on every younger pupil and misses it on every older one.

The inputThe actual education plans

The corpus is 52 education plans, 0.12 MB (txt 52). Plain text, one format, invented rather than fetched -- and on this subject that is not a convenience, it is the only defensible choice. A real education plan names a child, their family, their school and the professionals working with them; it is the last document a public repository should carry, and there is no public corpus of (plan, correct completeness verdict) pairs for a reason that is not an accident. Generating it also makes the label mechanical: gold's plan_outcome is the rulebook lookup over the same seven states the generator built the text from, never somebody's reading of the checklist.

Synthetic Record
----------------
GENERATED SAMPLE -- not a real pupil, school or district. No names, no dates and no diagnosis appear anywhere in this file.
Written by tools/build_corpus.py from seed 20260822.

Plan
----
PLN-BF-31487

Pupil Reference
---------------
PUP-L96-9836

Pupil Age
---------
16

Plan Status
-----------
draft

Present Levels of Performance
-----------------------------
Mathematics problem solving: 9 of a possible 16 on the autumn benchmark, against a year-group median of 14.
This level means the pupil cannot complete unmodified grade-level tasks in this area without the support set out below.

Annual Goals
------------
Goal 1 -- reading comprehension
  Baseline: 5 of 10 on the classroom rubric
  Criterion: 8 of 12 on the classroom rubric, on 4 of 5 occasions
  Measured by: a weekly curriculum-based measurement probe
Goal 2 -- written expression
  Baseline: 3 of 8 on the classroom rubric
  Criterion: 9 of 12 on the classroom rubric, on 4 of 5 occasions
  Measured by: a monthly timed fluency probe
Goal 3 -- organisation and task initiation
  Baseline: 4 of 10 on the classroom rubric
  Criterion: 6 of 8 on the classroom rubric, on 4 of 5 occasions
  Measured by: scored classroom work samples

Progress Measurement and Reporting
----------------------------------
Progress against each goal is measured by a fortnightly running record.
Progress will be shared as needed.

Services
--------
Occupational therapy -- 2 sessions per week, 25 minutes each, in a small-group room.
Specialist reading instruction -- 2 sessions per week, 25 minutes each, in the general classroom.
Assistive technology support -- 1 session per week, 25 minutes each, in the therapy room.

Accommodations
--------------

The outcomeWhat a good result looks like

A fourteen-field extracted record per plan -- a state for every component in the shipped rulebook, plus the two decoys copied verbatim -- and one pure-code worklist decision taken from two of those fields: a plan that is not complete, and is already in effect, is the row to open today. The answer names which components are missing, which are present and cannot be measured against, and what it could not determine. Nothing here approves, signs, files or amends a plan, and nothing here is a decision about a pupil.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Screening a queue of plans before a quality or programme lead opens them — one model call per plan -- it caught 30 of the 32 components that are present and cannot be measured against, where a checkbox review caught 0
  99.45 pct component-state accuracy against the free floor's 87.36 pct, and 0 false defects against the floor's 14. The difference is entirely in the half of the job that is inside the section rather than in the heading.
• Deciding which plans in a live caseload to open today — the worklist flag, and read the component list under it rather than the outcome
  20 of 20 flagged with no false alarms, against the floor's 8 of 20 -- and the floor's twelve misses are all plans already in effect whose only defect is unmeasurable rather than absent.
• Deciding whether this shape is worth buying at all, from these numbers — the component-state numbers, and nothing else on the page
  One tier, one run, 52 plans, one seed and one layout. What the numbers DO support is narrow and real: a checkbox review is measurably blind to the half of a completeness check that matters, and one model call over an invented rulebook is not. What they do not support is any claim about which model, or about a plan that is not a plain-text file with recognisable headings.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-iep-complete. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts, the 52-plan picker populates, the field table draws its fourteen empty rows, the four component states render with what getting each one wrong costs, and the shipped rulebook renders in full from data/rulebook.json; clicking Check returns the no-key sentence rather than a stack trace, and the worklist row reads “not computed -- one of the two values the rule needs was missing” rather than “no”. python -m evals.check_labels passes with no network access at all, and python -m evals.run --baseline scores the free floor with no key.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per education plan, measured around the adapter call with 12 concurrent workers

Current processWhat it replaces

Somebody opening each education plan, holding a list of the components the rules where they work require, and reading the plan section by section against it. The part that is quick is ticking off the headings. The part that takes the time, and the part that gets skipped when the meeting is tomorrow, is reading INSIDE each section to see whether what is written there can actually be measured against at the next review -- whether every goal states a baseline, a criterion and a method, whether every service line states its frequency, duration and location, whether the reporting schedule is a schedule rather than the word 'regularly'. And there is one component whose requirement depends on the pupil rather than the plan, so a reviewer who ticks the same seven boxes on every plan raises a false defect on every younger pupil and misses it on every older one.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 0 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings; a plan with none falls back to one whole-document segment

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only -- 52 plans cut into 745 sections in under six thousandths of a second, in process, with no model and no network. The section count is not 52 x 14 because a plan MISSING a component has no section for it, which is the whole subject of this kit.

LicenceLicence

MIT -- this repository's own licence. ⚠︎ NO REAL PUPIL DATA AND NOTHING RESEMBLING IT: there are no names anywhere (a pupil is a reference code), no dates and no dates of birth, no disability category, no diagnosis and no eligibility label tied to anybody. Every school and district name is an invented compound and no register of schools was consulted. Every plan opens with a Synthetic Record banner naming the script and seed that wrote it, so a single file copied out of this repository still says what it is. Nothing was fetched, scraped, anonymised or adapted from anywhere. The component rulebook this kit ships (data/rulebook.json) is INVENTED and is not an authority -- it reproduces no statute, regulation, state or district plan template, agency guidance or published checklist, and its transition age of 14 is a number this kit chose.

Bring your ownBring your own education plans

REPLACE data/rulebook.json FIRST. It is this kit's own construction, it reproduces no rule anybody wrote, and everything downstream reads it: src/rulebook.py loads it, src/prompt.py renders it into every call, and tools/build_corpus.py writes gold with it. Then replace data/corpus/*.txt, write your own data/fields.json, and supply a gold record per plan. SECTION_HINTS in src/select.py maps fields to section headings and WILL need editing -- and read that module's note before you do, because this kit deliberately drops the per-field whole-document fallback every sibling kit has. compute() in src/extract.py is this kit's own worklist rule and is the second thing to replace.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 3,594 tokens assembled, 1,790 are contexts — 50% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on IEP-0016 -- the same plan the worked example above was captured on, so the five parts sum to exactly the 3,594 input tokens that call actually reported. Five calls at max_tokens=1, each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. kits/UC0057-iep-complete/results/tokens-p002-iep-complete.json. ⚑ THE SAME MEASUREMENT ON A DIFFERENT PLAN (IEP-0001, tokens-p001-iep-complete.json) returned 907 / 983 / 241 / 897 for the first four parts -- byte-identical, which is the evidence that the 3,028-token fixed prefix really is fixed and is paid on every call.

Full promptThe prompt in full

You read an INDIVIDUALISED EDUCATION PLAN and check it against a component rulebook. You return JSON and nothing else.

You are producing a REVIEWER'S WORKLIST for a person to work through. You never approve, sign, file or amend a plan, and nothing you return is a decision about a pupil. The team that writes the plan decides what goes in it.

RULES, in order of importance:
1. If the plan does not state a field, return null for it. Do not infer it and do not use what you know about how plans are usually written.
2. Each of the seven COMPONENT fields is decided ONLY by the COMPONENT RULEBOOK given below, applied to the section the rule names. Work through these four questions IN ORDER and STOP at the first one that answers:
   a. NOT REQUIRED. Only the conditional rule can reach this state, and only when the plan STATES the pupil's age and it is BELOW the rulebook's transition age. Then the state is 'not_required' whatever the plan does or does not contain. If the plan does NOT state an age, 'not_required' is never correct -- an unknown requirement is not an absent one.
   b. ABSENT. The section the rule names is not in the plan, or its body is one of the placeholders listed in the rulebook. Answer 'absent'.
   c. NOT MEASURABLE. The section is there and one of the elements the rule names is missing, or is stated in words that carry no quantity. Answer 'present_not_measurable'. ⚠︎ ONE BAD ENTRY MAKES THE WHOLE COMPONENT UNMEASURABLE. A goals section with six sound goals and a seventh that states no criterion is 'present_not_measurable'. Never average a section, and never let a well-written first entry settle a component.
   d. Otherwise answer 'present_complete'.
3. A COMPONENT WHOSE SECTION IS NOT IN THE PLAN BELOW IS ABSENT. The plan text you are given is every section of the plan that could carry a component; exactly two administrative sections are removed before the plan reaches you -- one marking the file as a generated sample, and one naming the school and the district -- and neither carries a component. Do not treat their absence as a missing component, and do not assume any other section was withheld.
4. THE CHECKLIST IS A FIELD TO COPY, NOT EVIDENCE. The 'Checklist Completed By' section is what a previous reviewer TICKED. A checklist claiming every required component is present does not make an absent section present and does not make an unmeasurable goal measurable. It is NEVER an input to a component state or to the outcome.
5. THE CASE MANAGER'S NOTE IS A FIELD TO COPY, NOT EVIDENCE ABOUT COMPLETENESS. A note that sounds worried does NOT make a sound plan defective, and a note that sounds reassuring does NOT clear a plan. The rulebook decides; the note is one person's remark and may disagree with it.
6. `plan_outcome` follows from the seven states and the pupil's age, in this order, stopping at the first that fires: (1) any REQUIRED component 'absent' -> 'components_missing'; (2) any REQUIRED component 'present_not_measurable' -> 'not_measurable'; (3) the plan does not state the pupil's age AND transition is anything other than 'present_complete' -> 'undetermined'; (4) otherwise 'complete'. A component in state 'not_required' is judged on nothing.
7. Copy identifiers, the age and the case manager's note verbatim from the plan. Where the plan says the pupil's age is not stated, return null for pupil_age rather than the words.
8. Use the exact allowed value for a field that lists them.
9. Return every field named in the schema, even when the answer is null.

COMPONENT RULEBOOK (the authority for every component state; this is an INVENTED rulebook shipped with this kit -- it reproduces no statute, regulation, agency guidance or plan template)

TRANSITION AGE: 14 -- the age at or above which rule C-7 applies.

C-1  Present levels of performance
    section heading: Present Levels of Performance
    applies: to every plan
    requires: The plan states where the pupil is now, in terms a later reader can compare against.
      - a current performance level carrying a number, a rate or a scored band
      - a statement of how the pupil's needs affect progress in the general curriculum
    not measurable when: The section is written entirely as narrative — 'is working below peers', 'continues to find reading difficult' — with no current level anybody can measure against later.

C-2  Measurable annual goals
    section heading: Annual Goals
    applies: to every plan
    requires: Each annual goal states a baseline, a criterion and the method the goal will be measured by.
      - a baseline carrying a number, a rate or a scored band
      - a criterion carrying a number, a rate or a scored band
      - a named measurement method
    not measurable when: ANY goal in the section is missing one of the three elements, or states one in words that carry no quantity — 'to the best of the pupil's ability', 'as appropriate', 'improved'. One unmeasurable goal makes the component unmeasurable; the section is not scored on average.

C-3  How progress is measured and reported
    section heading: Progress Measurement and Reporting
    applies: to every plan
    requires: The plan states how progress against the goals will be measured AND how often it will be reported to the family.
      - a named measurement method
      - a stated reporting frequency
    not measurable when: The section names a method and never says how often anybody will be told — or says 'regularly', 'as needed', 'periodically', none of which is a frequency.

C-4  Services, with frequency, duration and location
    section heading: Services
    applies: to every plan
    requires: Each service the plan commits to states its frequency, its duration and where it is delivered.
      - a frequency for every service line
      - a duration for every service line
      - a location for every service line
    not measurable when: ANY service line is missing one of the three. A service named with no frequency cannot be reconciled against what was actually delivered.

C-5  Accommodations, and where each applies
    section heading: Accommodations
    applies: to every plan
    requires: The plan names specific accommodations and states, for each, the setting or activity it applies to.
      - at least one named, specific accommodation
      - a setting or activity for every accommodation named
    not measurable when: The section lists accommodations with no setting, or is written as a general permission — 'accommodations as needed', 'support as required', 'teacher discretion'.

C-6  Participation with peers, and the rationale
    section heading: Participation with Peers
    applies: to every plan
    requires: The plan states the extent to which the pupil will NOT take part with peers who have no plan, and why.
      - a stated extent, carrying a number or a proportion of the school day
      - a reason for that extent
    not measurable when: The section states an intention with no extent — 'will participate with peers as appropriate', 'included wherever possible' — or gives an extent with no reason attached to it.

C-7  Transition content
    section heading: Transition
    applies: Required only when the plan states the pupil's age and it is at or above the rulebook's transition age. Below that age the component is `not_required`, whatever the plan does or does not contain. When the plan does NOT state the pupil's age, the requirement cannot be determined — describe what the document contains and let the outcome rule report the gap.
    requires: The plan states post-school goals and the steps or services that lead to them.
      - at least one post-school goal
      - the steps or services that lead to it
    not measurable when: The section states an aspiration with no steps under it — 'will explore options after school' — so nothing in the plan year moves the pupil towards it.

A SECTION BODY THAT IS ONLY A PLACEHOLDER IS ABSENT, NOT PRESENT. Treat these as placeholders: 'to be completed'; 'to be completed at the meeting'; 'n/a'; 'see pupil file'; 'see file'; 'tbc'.

COMPONENT STATES -- answer every component with exactly one of these four
  present_not_measurable
      The component is in the plan and cannot be measured against. An element the rulebook asks for is missing, or is written in words that carry no quantity -- a goal with no baseline, a service with no frequency, a reporting schedule that says 'regularly'. ONE bad entry makes the whole component unmeasurable; the section is never scored on average.
  absent
      The plan has no section for this component, or the section is a placeholder -- 'to be completed', 'see pupil file'. There is nothing to read and nothing to quote.
  present_complete
      Every element the rulebook names for this component is in the plan, and each one is stated in terms somebody can measure against at the next review.
  not_required
      The rulebook does not ask this plan for this component. Only the transition component can be in this state, and only when the plan STATES the pupil's age and it is below the rulebook's transition age. When no age is stated, this state is never correct -- an unknown requirement is not an absent one.

Extract these fields:
- plan_id (string) -- the plan identifier, verbatim
- pupil_ref (string) -- the pupil reference code, verbatim. It is a code, never a name — this corpus contains no names
- pupil_age (integer) -- the pupil's age in whole years, as the plan states it. When the plan says the age is not stated, return null -- do not estimate it, do not infer it from anything else on the plan, and do not copy the words
- plan_status (enum) one of: draft, in_effect -- is this plan still a draft, or is it already in effect?
- present_levels (enum) one of: absent, present_not_measurable, present_complete -- rule C-1. The state of the Present Levels of Performance section: absent if there is no such section or its body is a placeholder; present_not_measurable if it is there and is missing one of C-1's elements; present_complete otherwise
- annual_goals (enum) one of: absent, present_not_measurable, present_complete -- rule C-2. The state of the Annual Goals section. present_not_measurable if ANY goal in it is missing a baseline, a criterion or a measurement method, or states one in words carrying no quantity. One unmeasurable goal makes the whole component unmeasurable -- the section is never scored on average
- progress_measurement (enum) one of: absent, present_not_measurable, present_complete -- rule C-3. The state of the Progress Measurement and Reporting section. present_not_measurable if it names a method and no reporting frequency, or gives a frequency that is not one ('regularly', 'as needed', 'periodically')
- services (enum) one of: absent, present_not_measurable, present_complete -- rule C-4. The state of the Services section. present_not_measurable if ANY service line is missing its frequency, its duration or its location
- accommodations (enum) one of: absent, present_not_measurable, present_complete -- rule C-5. The state of the Accommodations section. present_not_measurable if an accommodation is named with no setting, or the section is a general permission ('as needed', 'teacher discretion')
- participation_rationale (enum) one of: absent, present_not_measurable, present_complete -- rule C-6. The state of the Participation with Peers section. present_not_measurable if it states an intention with no extent, or an extent with no reason
- transition (enum) one of: absent, present_not_measurable, present_complete, not_required -- rule C-7, the CONDITIONAL one. Answer 'not_required' when the plan STATES the pupil's age and it is BELOW the rulebook's transition age -- whatever the plan does or does not contain. Otherwise describe the document: absent, present_not_measurable or present_complete. When the plan does NOT state an age, never answer not_required -- an unknown requirement is not an absent one
- checklist_claim (enum) one of: all_present, items_outstanding, not_completed -- what the Checklist Completed By section CLAIMS a previous reviewer found, copied as stated. It is a field to report and it is NEVER an input to any component state or to the outcome
- case_manager_note (string) -- the case manager's own free-text remark on the plan, copied verbatim
- plan_outcome (enum) one of: complete, components_missing, not_measurable, undetermined -- the outcome the seven component states and the pupil's age produce under the shipped rulebook. Work through the four checks IN ORDER, stopping at the first that fires. (1) Any REQUIRED component 'absent' -> components_missing. (2) Any REQUIRED component 'present_not_measurable' -> not_measurable. (3) The plan does not state the pupil's age AND transition is anything other than present_complete -> undetermined. (4) Otherwise -> complete. A component whose state is 'not_required' is judged on nothing. Never decide this from the checklist's claim and never from the case manager's note

Return a JSON object with exactly these keys: plan_id, pupil_ref, pupil_age, plan_status, present_levels, annual_goals, progress_measurement, services, accommodations, participation_rationale, transition, checklist_claim, case_manager_note, plan_outcome
Use null for any field the plan does not state.

EDUCATION PLAN
--------------
Plan
----
PLN-DN-60887

Pupil Reference
---------------
PUP-C61-3660

Pupil Age
---------
9

Plan Status
-----------
in_effect

Present Levels of Performance
-----------------------------
Reading comprehension: 8 of a possible 15 on the autumn benchmark, against a year-group median of 12.
This level means the pupil cannot complete unmodified grade-level tasks in this area without the support set out below.

Annual Goals
------------
Goal 1 -- mathematics problem solving
  Baseline: 5 of 11 on the classroom rubric
  Criterion: 9 of 12 on the classroom rubric, on 4 of 5 occasions
  Measured by: a monthly timed fluency probe
Goal 2 -- reading comprehension
  Baseline: 2 of 10 on the classroom rubric
  Criterion: 8 of 12 on the classroom rubric, on 4 of 5 occasions
  Measured by: the school's own writing rubric
Goal 3 -- attention and self-regulation
  Baseline: 2 of 8 on the classroom rubric
  Criterion: 6 of 12 on the classroom rubric, on 4 of 5 occasions
  Measured by: scored classroom work samples

Progress Measurement and Reporting
----------------------------------
Progress against each goal is measured by the school's own writing rubric.
Progress is reported to the family every 12 weeks, alongside the general report card.

Services
--------
Specialist reading instruction -- 4 sessions per week, 25 minutes each, in the resource room.
Mathematics intervention group -- 3 sessions per week, in a small-group room.
Speech and language therapy -- 3 sessions per week, 25 minutes each, in the resource room.

Accommodations
--------------
Text-to-speech for reading passages.
A printed copy of every set of instructions -- in every subject.

Participation with Peers
------------------------
The pupil will be outside the general classroom for 360 minutes of the 1,650 minutes in a school week, for the services set out above, because that instruction cannot be delivered at the intensity needed in a class of 29.

Checklist Completed By
----------------------
Reviewer R-6509 -- all required components present: yes

Case Manager Note
-----------------
Straightforward plan, nothing unusual about it at all.

Raw responseThe raw response

{"plan_id":"PLN-DN-60887","pupil_ref":"PUP-C61-3660","pupil_age":9,"plan_status":"in_effect","present_levels":"present_complete","annual_goals":"present_complete","progress_measurement":"present_complete","services":"present_not_measurable","accommodations":"present_complete","participation_rationale":"present_complete","transition":"not_required","checklist_claim":"all_present","case_manager_note":"Straightforward plan, nothing unusual about it at all.","plan_outcome":"not_measurable"}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch education plans that pass review but can't be measured — 52 education plans. One model answered, and every answer was then graded Four different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran these runs publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersFour ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The checkbox floor (evals/baseline.py) and the scorer (evals/judge.py) are both pure code and cost $0.00 to run against any result set. The figure above is the ONE scored run's own token counts (52 plans) priced at Google Gemini 3 Flash's published rate -- the same basis cost_per_query_usd uses. ⚠︎ IT IS NOT WHAT BUILDING THIS KIT COST. Three further paid passes sit outside it and are committed in results/: a 6-plan calibration that MEASURED max_tokens ($0.0315), ten prompt-split calls at max_tokens=1 across two plans ($0.0116), and one worked example plus one live UI screenshot ($0.0097). All 70 live calls together project to $0.3257 on this card.

Cost driversWhat actually moves the bill

• The fixed prefix -- system prompt (907 tokens), component rulebook (983), the four state definitions (241) and the field schema (897) -- is 3,028 of 3,594 tokens on the example call, 84 pct, and is byte-identical on every one of the 52 calls. The plan itself is 566 tokens. The bill is the rulebook and the instructions, not the document.
• Provider-side reasoning. 51,994 of the run's 59,942 output tokens (86.7 pct) were reasoning rather than the JSON record, on a fourteen-key reply that is about 150 tokens of actual content. Reasoning was left at the provider's default and nothing here has measured what turning it off would cost in accuracy -- and on a kit whose hard half is a judgement about the inside of a section, that is a more interesting question than it is on a pure extraction kit.
• Output length: the model returns a full fourteen-key JSON record every call, including the case manager's note copied back verbatim, whatever the plan says.

Your volumeWhat it costs at your volume

Linear in plans: each call is independent, carries the same fixed prefix and shares nothing with its neighbours, so 520 plans cost ten times 52. Wall clock is not linear in the same way -- the run took 46.4 seconds at 12 concurrent workers rather than the ~7.5 minutes it would take serially, and EVAL_WORKERS is the knob. Nothing amortises: there is no index and no cache. ⚠︎ AND THE RULEBOOK DOES NOT SCALE LINEARLY, IT SCALES WORSE: it is sent whole in every call, so a real authority's checklist of forty or eighty required elements, each with its own definition and its own measurability test, multiplies the input side of every call at once. At that size the rulebook has to be split -- only the rules relevant to the sections actually present sent per call -- and this kit's prompt would have to be rebuilt around it.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 52 education plans, one completion call each, on one tier. There is no second tier on this kit and therefore no second row.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Six of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the plan into addressable sections on its underlined headings, pure code

# Cut a document into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections carry each field, pure code -- the outcome is mapped to the seven component sections and the pupil's age, never to the previous reviewer's checklist and never to the case manager's note; the Synthetic Record banner and the School and District section are mapped by nothing and never sent. ⚠︎ AND IT HAS NO PER-FIELD FALLBACK, unlike every sibling kit: an absent section is the finding here, so falling back to the whole document when a section is missing would fire on half the corpus

You change it to: map fields to your own plan's headings. ⚠︎ READ THE MODULE NOTE BEFORE YOU CHANGE IT -- this kit deliberately has no per-field fallback to the whole document, because an absent section is the finding rather than a lookup failure

# Pick which sections plausibly carry each field. Pure code -- the last deterministic step before

SECTION_HINTS = {
SECTION_HINTS["plan_outcome"] = ["Pupil Age"] + [SECTION_BY_KEY[k] for k in COMPONENTS]
NEVER_SENT = ["Synthetic Record", "School and District"]
def for_field(secs, field):
def plan(secs, fields):

src/rulebook.pyrulebook

the component lookup, loaded from data/rulebook.json -- seven components, the elements each one needs to be measurable, the conditional rule and its age, and the four ordered checks that turn seven states into one plan outcome. Data, not code, so a forker can open it and disagree with it

# THE COMPONENT RULEBOOK, loaded from data/rulebook.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
def load(path=RULEBOOK_PATH):
M = load()
RULES = list(M["rules"])
COMPONENTS = [r["key"] for r in RULES]
RULE_BY_KEY = {r["key"]: r for r in RULES}
SECTION_BY_KEY = {r["key"]: r["section"] for r in RULES}
STATES = tuple(M["states"])

src/prompt.pyprompt — a swap seam

assemble one call for all fourteen fields, with the rulebook RENDERED into it from data/rulebook.json rather than restated in prose -- a list of required elements the model has never been shown is not a list. Also the single declaration of the component-state vocabulary, which the parser, the scorer and the published app panel all read from here

You change it to: the component-state vocabulary and what getting each one wrong costs. It is declared once and read by the prompt, the parser, the scorer and the published app panel; a second copy in a data file is the thing that drifts from the scorer

# Assemble the completeness-check prompt. One prompt per plan, all fourteen fields in it.

VERDICTS = ("present_not_measurable", "absent", "present_complete", "not_required")
VERDICT_MEANINGS = {
OUTCOMES = ("complete", "components_missing", "not_measurable", "undetermined")
OUTCOME_MEANINGS = {
def _state_block():
def rulebook_block(m=None):
RULEBOOK_TEXT = rulebook_block()
STATE_TEXT = _state_block()
SYSTEM = (

src/extract.pyextract — a swap seam

the AI layer, one provider one key -- plus the pure-code business-condition check downstream: a plan that is not complete, and is already in effect, goes on the worklist

You change it to: the worklist rule itself -- this kit ships two values (not complete AND already in effect) and a real review queue weighs how long the plan has left to run, which component is missing, whether the next meeting is booked and who can reopen it. It is deliberately NOT the same function as the rulebook lookup, so changing WHAT GOES ON THE WORKLIST does not change WHAT COUNTS AS COMPLETE

# Check one education plan: segment, select, prompt, one model call, then a pure-code

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 5000
def load_fields():
def load_doc(plan_id):
def documents():
def decide(values):
def correct_outcome(values):

evals/judge.pyjudge

score the fourteen fields, the seven component states, the two directions of component error (waved through / falsely flagged), the four-way plan outcome and the worklist flag, all separately, pure code

# Score a completeness-check run. PURE CODE -- gold is exact and the answer is one value per cell,

CLEARED = ("present_complete", "not_required")
NEEDS_WORK = ("absent", "present_not_measurable")
def norm(v):
def equal(field, got, want):
def score(fields, records, golds):
def _matrix(rows, positive):
def score_verdicts(records, flags, golds):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3578 input and 1152 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's plans are entirely synthetic (tools/build_corpus.py, seed 20260822) and the corpus is built so that it could not be mistaken for anything else: no names anywhere, no dates and no dates of birth, no disability category, diagnosis or eligibility label, invented school and district names, and a Synthetic Record banner as the first section of every file. Nothing was fetched, scraped or anonymised from anywhere. The only outbound traffic the kit makes is one chat-completion request per plan to the configured provider, carrying the mapped sections of one plan plus the fixed prompt and the shipped rulebook -- and the Synthetic Record banner and the School and District section are mapped by no field and never leave the machine, which evals/check_labels.py asserts on every plan before a run may spend. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser.

The experimentWe did not attack it — and three of four boundaries hold

The three boundaries that hold were confirmed by reading the code and the corpus generator, not by a run: no code path approves, signs, files or amends a plan; there is no real pupil data anywhere because nothing was ever fetched to anonymise; and the rulebook lookup and the worklist rule read only enums and an integer, so neither decoy can reach either. The fourth is open. An indirect prompt injection needs a field an outside party authored, and this kit has two -- the previous reviewer's checklist and the case manager's note -- which makes them the obvious place to attack and the reason the gap is named rather than glossed. Confirmed by reading the code and the corpus generator, not by a run, on 2026-08-22.

The first three boundaries hold, the first and third confirmed by reading the code and the second by the corpus generator having no input to take data from. The fourth is open and is named as open -- an indirect prompt injection in the case manager's note is exactly the attack this kit's shape invites, and nothing here has tried it.

The result0 attack trials, and three of four boundaries checked here hold on evidence. The one that does not is the one this kit's own shape invites: the checklist and the case manager's note are free text a third party wrote, and whether an instruction hidden in either could move a component state is unmeasured.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted ambiguity is a REGISTER mismatch and a contradicting tick-box -- a reassuring note on a defective plan, a checklist claiming every component is present when one is not -- which measures whether a model runs the rulebook when the page points the other way. It does not measure whether a model obeys an instruction hidden in the same field. Those are different failures and only one of them is measured here.

HonestyWhat this does not prove

• Whether an indirect prompt injection in the case_manager_note or checklist_claim field could move a component state or the outcome. No attack run exists.
• Whether the kit behaves safely against a hostile provider -- a response body crafted to break the JSON extraction in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed document, and nothing beyond that was tested.
• Whether a real plan would carry anything this kit mishandles. The corpus has no personal data by construction; a real plan carries a named child, a family, a school and a professional's own assessments, and none of that path is exercised here at all.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

on_worklist fires when the plan is NOT complete AND it is already in effect -- plan_outcome != "complete" and plan_status == "in_effect". Both values come out of the same reply; the rule is run afterwards in pure code, over whatever the model returned, and never over gold. `undetermined` counts as not-complete, deliberately: a plan in effect whose transition requirement nobody can determine is precisely a row to work. A reply missing either value returns None rather than False -- an unknown is not a pass.

src/extract.py::compute(), called by extract() on every plan, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by identical code. The completeness lookup it reads lives in a SEPARATE module, src/rulebook.py::decide(), which is also what the corpus generator wrote gold with and what src/prompt.py states to the model -- one definition, three readers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT an approval, a rejection or an amendment. Nothing here signs, files, submits or changes a plan, notifies anybody or writes to any system, and no component state is a decision anybody is bound by. The team that writes the plan decides what goes in it.
• It is NOT a judgement about a pupil. There is no eligibility field, no placement field, no diagnosis and no category anywhere in this kit, and the corpus carries none by construction. A completeness check that started answering those questions would be a different and much more dangerous tool.
• It is NOT a check on whether the component states are right. If the model reads a section wrongly and then computes the outcome correctly from that reading, the plan goes on the worklist -- or does not -- on a wrong fact, and this rule cannot tell. ⚠︎ AND THE CONSISTENCY DIAGNOSTIC IS BLIND TO THE SAME CASE, MEASURED RATHER THAN ASSUMED: it caught 0 of the free floor's 20 outcome errors, because a checkbox review is perfectly self-consistent.
• It is NOT anybody's review policy. Not-complete-and-already-in-effect is this kit's own simplification, invented for this corpus. No district procedure, agency guidance or regulation was consulted, and none is reproduced. A real queue weighs how long the plan has left to run, which component is missing, whether the next meeting is booked and who has authority to reopen it.
• It is NOT a judgement about the rulebook. The flag reads the outcome the rulebook produced; if the rulebook is wrong, the flag is confidently wrong with it. The shipped rulebook is invented and is not an authority.
• It is NOT a measure of plan QUALITY. A plan can be perfectly complete and completely wrong for the pupil. Measurability is a floor, not a standard, and nothing in this kit has a view on whether a measurable goal is an appropriate one.

WatchedWhat is watched, and why that one

The kit measures 14 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 7 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 52, extraction_cells 728, failures 0, refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

extraction · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on documents, extraction_cells, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

extraction · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 7 chips that all say so.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/extract.py, on any change to data/rulebook.json (which moves the outcome the flag reads), on any change to data/fields.json's allowed values for plan_outcome or plan_status, and on any corpus regeneration. Changing WHAT GOES ON THE WORKLIST is a policy change and must be re-scored, even though it never changes what counts as complete.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the rulebook is a JSON file read with json, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per plan -- segment, select, prompt, one call, parse, compute -- with no branch, no loop and no agent. Anything that looks like orchestration in a kit this size is a diagram of a straight line.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain -- the JSON extraction from a fenced reply, the backoff policy, the section regex, the per-component text parsers in the corpus verifier and the .env reader are all code somebody has to own.
• No framework means no framework's ecosystem -- no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. What ships is what is in the repository.
• The provider abstraction covers exactly two wire formats. A third provider is one function and one dict entry, and until somebody writes it the kit runs on two shapes.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-iep-complete on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b000-iep-complete-checkbox, t000-iep-complete-stub recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence. ⚠︎ NO REAL PUPIL DATA AND NOTHING RESEMBLING IT: there are no names anywhere (a pupil is a reference code), no dates and no dates of birth, no disability category, no diagnosis and no eligibility label tied to anybody. Every school and district name is an invented compound and no register of schools was consulted. Every plan opens with a Synthetic Record banner naming the script and seed that wrote it, so a single file copied out of this repository still says what it is. Nothing was fetched, scraped, anonymised or adapted from anywhere. The component rulebook this kit ships (data/rulebook.json) is INVENTED and is not an authority -- it reproduces no statute, regulation, state or district plan template, agency guidance or published checklist, and its transition age of 14 is a number this kit chose. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-field exact match, light normalisation

Does the model's value for each of the fourteen fields match gold, after trimming whitespace and punctuation? pupil_age is the one legitimately-null field -- null exactly where the plan says the age is not stated -- so a null there is a hit, not a miss, when gold agrees.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every call, and the same field-match logic the free floor is scored by -- a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 52 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same values the plan states; evals/check_labels.py asserts every non-nullable field is populated on every row, that every gold outcome is its own rulebook lookup, that transition=='not_required' agrees with the stated age in BOTH directions, and that each of the four hard cases clears its own floor, before any run is allowed to spend.

How tight can the band be? There is no tolerance band -- it is exact match after trimming whitespace and punctuation, never a continuous score to round.

Cadence: Re-score on any change to data/rulebook.json, src/prompt.py, data/fields.json or tools/build_corpus.py -- the first three change what is asked, the fourth changes what is asked about. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe component state, and the two directions of being wrong

Two questions at once, over every (plan, component) pair. Exact: did the run land on the same state the rulebook produces -- absent, present_not_measurable, present_complete or not_required? And the two that are NEVER averaged together: PASSED UNMEASURABLE, a component that is present and cannot be measured against which the run cleared, and FALSE DEFECT, a sound component the run flagged. The first sends a plan to a meeting with a tick on a goal nobody can evaluate; the second puts a row on a person's worklist for nothing. They cost differently and an accuracy figure hides which one you are making.

MethodHow the test was run

evals/judge.py::score_verdicts, in-process, no key and no model.

Every grader on these pages scored the same 52 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's seven states are the values tools/build_corpus.py built the plan text from, and _verify() asserts every one of them is readable off that plan by a per-component parser -- a digit in the present-levels body, all three labelled elements in every goal block, a named location in every service line, a setting on every accommodation.

How tight can the band be? No threshold -- the state is one of four allowed values, and a reply returning none of them is counted as unanswered rather than folded into a correct cell.

Cadence: Re-run whenever data/rulebook.json or src/rulebook.py changes, whenever the corpus is regenerated, and on any provider or model change.

The decisionWhen to reach for it

In one lineThe plan outcome, four-way and collapsed onto needs-work

Did the run land on the same outcome the rulebook produces -- complete, components_missing, not_measurable or undetermined? And the one a reviewer acts on: OF EVERY PLAN SOMEBODY HAS TO OPEN, HOW MANY DID THE RUN FLAG? not complete is the positive class, and the expensive direction -- a plan the rulebook would have flagged that the run cleared -- is counted and named on its own rather than averaged into an accuracy figure where it disappears.

MethodHow the test was run

evals/judge.py::score_verdicts, in-process, no key and no model.

Every grader on these pages scored the same 52 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's outcome is re-derived inside the grader by the same lookup the kit publishes -- src/rulebook.py::decide(), reading data/rulebook.json -- so the truth this matrix grades against can never be a separately-typed label that drifted from the rule.

How tight can the band be? No threshold -- the outcome is one of four allowed values, and a reply returning none of them is counted as unanswered rather than folded into a correct cell.

Cadence: Re-run whenever data/rulebook.json or src/rulebook.py changes, whenever the corpus is regenerated, and on any provider or model change.

The decisionWhen to reach for it

In one lineon_worklist against the same rule run over gold

Does the pure-code worklist decision -- not complete AND already in effect -- land on the same plans it would land on if both fields had been read perfectly? It is a BUSINESS CONDITION, so unlike a self-consistency check it genuinely needs labels, and saying so is half of what makes the number believable.

MethodHow the test was run

evals/judge.py::score_verdicts, in-process, no key and no model.

Every grader on these pages scored the same 52 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute(), the same function the run uses, applied to GOLD's outcome and plan_status. One rule, two inputs, so a change to the rule moves both sides together and the grader cannot silently grade an old policy.

How tight can the band be? No threshold -- two values and a comparison.

Cadence: Re-check on any change to compute() in src/extract.py. Changing WHAT GOES ON THE WORKLIST is a policy change and must be re-scored, even though it never changes what counts as complete.

The decisionWhen to reach for it