The business caseThe problem this solves

An independence attestation register is a list of people, a list of returns and a list of relationships, and the question somebody has to answer from it every cycle is which of those people needs chasing today. The three cheap ways to get it wrong are all on the page: a person who has left the engagement, a person who joined last week and a role that never carried a requirement all have no return on file and none of them owes anything; a return filed a week after the due date is inside the grace window and is in time; and a relationship the register records as disposed of is only a contradiction if the disposal came BEFORE the return that declares it. Miss any of those and the queue fills with people who are fine, which is how a monitoring queue stops being read. Somebody opening an engagement's independence attestation register, working down the roster, and for each person deciding four things in the right order: whether they owe an attestation at all, when it was due — which no register states, so it has to be worked out from the date their cycle opened and the cycle length for their role — whether the return on file was in time and about the right window, and whether anything else on the register contradicts what it declares. It is a few minutes per engagement and it is every engagement, and the part that goes wrong is not the arithmetic: it is chasing the three people who owed nothing, and clearing the person whose two returns disagree because a form exists.

The inputThe actual attestation registers

The corpus is 50 attestation registers, 0.08 MB (txt 50). Plain text, one format, invented rather than fetched — a real independence register names real people on a real engagement, their real financial and family relationships, a real client and a named administrator's own remark, and there is no public corpus of (attestation register, correct worklist) pairs for the same reason there is no public corpus of HR files. Generating it also makes the label mechanical: gold's status AND gold's derived due date are both the rulebook run over the same values the register states, never somebody's reading of the administrator's note.

Engagement
----------
ENG-6826-AH

Register As At
--------------
2026-02-26

Cycle Rulebook
--------------
IAR-1 -- illustrative, shipped with this kit; reproduces no real professional standard

Attesters On Record
-------------------
P-7868   quality_reviewer         cycle opened 2025-05-28
P-5176   specialist_advisor       cycle opened 2024-12-07
P-6804   senior_manager           cycle opened 2025-01-12
P-2667   specialist_advisor       cycle opened 2025-01-10

Returns Filed
-------------
P-6804   filed 2026-01-08   covering the period to 2026-01-12   declared: no interests to report
P-2667   filed 2025-12-05   covering the period to 2026-01-11   declared: no interests to report
P-7868   filed 2025-09-30   covering the period to -- not stated on the filed return   declared: family member employed by a supplier of the client

Holdings And Relationships On File
----------------------------------
P-7868   family member employed by a supplier of the client -- current, no disposal recorded

Roster Changes
--------------
P-5176   role vacated 2026-01-12 -- the requirement no longer applies

Register Notes
--------------
Declarations on this engagement are under manual review; treat the register as provisional.

Prepared By
-----------
engagement administration, register assembled 2026-02-26

The outcomeWhat a good result looks like

A four-field register record plus one object per person on the roster, and a status per person taken afterwards in pure code: satisfied, missing, stale, contradicted, not required, or not determinable from this register. Each carries the rulebook's own reasoning and the derived due date the code computed. One register-level routing flag says whether this engagement needs somebody to open the file rather than send a reminder. Nothing here chases anybody, files anything, signs a register off or clears a person.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Turning a book of independence registers into a worklist somebody will actually read — the fast tier — 0 false alarms on 101 flagged rows, against a box-tick floor's 32 on 62
  The number that decides whether a monitoring queue survives contact with a human is the share of its rows that did not belong there. It was 0.0 pct here and 51.61 pct on the floor, on the same 280 people. Missed breaches were 0 of 104 against the floor's 74.
• Deciding how much of the answer to let the model take — extraction only — let pure code decide the status
  The model was asked for its own status and its own due_on on every one of the 273 people it answered, and neither is published: the kit prints src/rulebook.py's. On this run the two agreed everywhere (0 replies disagreed with their own values), which is evidence the split costs nothing and is NOT evidence it is unnecessary — the split is what makes the date arithmetic measurable as arithmetic rather than as copying, and it is what a forker changing the rulebook edits in one place.

And where nothing here is good enough:

• Deciding whether this shape is worth buying at all, from these numbers — nothing yet — this corpus cannot answer it
  One tier, one run, 50 registers, and every figure on the 49 that answered is perfect. Nothing here separates two models, two prompts or two settings. What the numbers DO support is narrower and real: a box-tick monitor is measurably unsafe on this shape of decision, in BOTH directions at once — it chases people who owe nothing and clears people whose register contradicts itself — and the gap is entirely in the deciding half, because its reading is perfect.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-attestation-watch. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts on 127.0.0.1:8802, the 50-register picker populates, the four register rows and the roster panel draw their “not checked yet” state, and the shipped rulebook renders in full from data/rulebook.json; clicking Check returns the no-key sentence rather than a stack trace, and the routing row reads “—” rather than “no”. python -m evals.check_labels and python -m evals.run --baseline both pass with no network access at all.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per attestation register

Current processWhat it replaces

Somebody opening an engagement's independence attestation register, working down the roster, and for each person deciding four things in the right order: whether they owe an attestation at all, when it was due — which no register states, so it has to be worked out from the date their cycle opened and the cycle length for their role — whether the return on file was in time and about the right window, and whether anything else on the register contradicts what it declares. It is a few minutes per engagement and it is every engagement, and the part that goes wrong is not the arithmetic: it is chasing the three people who owed nothing, and clearing the person whose two returns disagree because a form exists.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings; a register with none falls back to one whole-document segment

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 50 registers cut into 450 sections in seven thousandths of a second, in process, with no model and no network. And there is no scheduler, no poller and no store either: this kit is called, it answers, and it holds nothing.

LicenceLicence

MIT — this repository's own licence. Every engagement reference, person reference, role, relationship description and administrator note is invented; no real firm, standard-setter, regulator, standard number, client or person is named anywhere in this kit. The cycle rulebook it ships (data/rulebook.json) is ILLUSTRATIVE and is not an authority. It was written for this kit and reproduces no real professional standard, standard-setter's rule, regulator's requirement or firm policy.

Bring your ownBring your own attestation registers

REPLACE data/rulebook.json FIRST. It is this kit's own construction, it reproduces no real standard or firm policy, and everything downstream reads it: src/rulebook.py loads it, src/prompt.py renders it into every call, and tools/build_corpus.py writes gold with it. Then replace data/corpus/*.txt, write your own data/fields.json — note it is TWO field sets, register-level and per-person — and supply a gold record per register carrying its whole roster. SECTION_HINTS in src/select.py maps fields to section headings and will need editing for a different register layout; when it does not match, selection falls back to the whole document — slower, more expensive, always correct. compute() in src/extract.py is this kit's own routing rule and is the second thing to replace.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 3,226 tokens assembled, 1,169 are schemas — 36% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on ATT-0032 — the same register the worked example above was captured on, so the four parts sum to exactly the 3,226 input tokens that call actually reported. Four calls at max_tokens=1, each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. kits/UC0053-attestation-watch/results/tokens-p003-attestation-watch.json. ⚑ THE SAME MEASUREMENT ON TWO OTHER REGISTERS (ATT-0001, tokens-p001; ATT-0044, tokens-p002) returned 1,002 / 566 / 1,169 for the first three parts on both — byte-identical, which is the evidence that the 2,737-token fixed prefix really is fixed and is paid on every call.

Full promptThe prompt in full

You read ONE engagement's INDEPENDENCE ATTESTATION REGISTER -- a snapshot of who is on the engagement, what each of them has filed, and what the register records about their declared relationships -- and you extract structured fields from it. You return JSON and nothing else.

You are PROPOSING a worklist for a qualified person to act on. You never chase anybody, never file anything, never sign a register off and never clear anyone. The rulebook below is an ILLUSTRATIVE rulebook shipped with this kit; it is not a professional standard and it is not an authority.

RULES, in order of importance:
1. If the register does not state a field, return null for it. Do not infer it and do not use what you know about the world.
2. Return one object per person listed in `Attesters On Record`, in the order that section lists them, and no others. Never invent a person, and never drop one because there is nothing else on the register about them.
3. `due_on` is DERIVED AND IS NEVER STATED ON THE REGISTER. Take the person's cycle-opened date and add the cycle length the rulebook gives for their role. Return null when the cycle-opened date is not recorded, and null when the rulebook gives no cycle length for the role.
4. `status` is decided ONLY by the RULEBOOK given below. Work through these five gates IN ORDER and STOP at the first one that fires:
   a. NOT REQUIRED. The Roster Changes section records this person's role as vacated, or records them as having joined inside the new-joiner window; or their role is one the rulebook puts no attestation requirement on. Answer 'not_required'. THIS GATE IS FIRST. All three of these people usually have no return on file, and treating that as a gap is the error this kit exists to measure.
   b. NOT DETERMINABLE. The register does not record when this person's cycle opened, so no due date can be derived; or a return is on file and the register does not state what period it covers, so the coverage test cannot be run. Answer 'not_determinable'. This is a real answer, not a failure to produce one -- a confident wrong 'fine' is the expensive mistake here.
   c. MISSING. No return is on file for this person at all. Answer 'missing'.
   d. CONTRADICTED. The person has more than one return and the earlier one declares something different from the later one; or the latest return declares a relationship the holdings section records as DISPOSED ON A DATE BEFORE THE RETURN WAS FILED. Answer 'contradicted'. A disposal recorded AFTER the filing date is NOT a contradiction -- the return was correct on the day it was filed.
   e. STALE. The latest return was filed later than the due date plus the rulebook's grace window; or it covers a period ending BEFORE the due date, which attests to the previous cycle. Answer 'stale'.
   Anything that survives all five gates is 'satisfied'.
5. THE SAME ROSTER LINE MEANS TWO DIFFERENT THINGS. 'cycle opened -- not recorded on this register' appears both on somebody who joined mid-cycle and on a record nobody wrote a cycle date for. The line itself cannot tell you which. ONLY the Roster Changes section can, and it is a different section of the register.
6. THE LATER FILING GOVERNS, AND THE RETURNS SECTION IS NOT SORTED. It is not grouped by person and not in date order. Read every line for a person before deciding which of them is their latest return; the earlier one is not discarded, it is compared against the later one.
7. THE ADMINISTRATOR'S NOTE IS A FIELD TO COPY, NOT EVIDENCE ABOUT ANYBODY'S STATUS. A note that sounds settled does NOT clear a gap, and a note that sounds worried does NOT create one. The register's own dates decide; the note is one person's remark and may disagree with them.
8. Copy references, roles and declarations verbatim from the register, in the case it states them. Every date is yyyy-mm-dd.
9. Use the exact allowed value for a field that lists them.
10. Return every field named in the schema for every person, even when the answer is null.

ATTESTATION RULEBOOK IAR-1 (the authority for `status` and `due_on`; this is an ILLUSTRATIVE rulebook shipped with this kit, not a professional standard)

ROLES THAT CARRY AN ATTESTATION REQUIREMENT
  engagement_lead, quality_reviewer, senior_manager, specialist_advisor

ROLES THAT CARRY NONE -- a person in one of these owes nothing, whatever the register lists
  engagement_administrator

CYCLE LENGTH, in days from the date the cycle OPENED to the date the attestation is DUE
  engagement_lead:         180 days
  quality_reviewer:        180 days
  senior_manager:          365 days
  specialist_advisor:      365 days

GRACE WINDOW
  14 days after the due date. A return filed inside it is IN TIME.

COVERAGE TEST
  The return must cover a period ending ON OR AFTER the due date. A return whose covered period ends before the due date attests to the wrong window: it is a real return, filed on time, about the previous cycle.

WHICH RETURN GOVERNS
  When one person has filed more than once, the return with the LATER filing date governs. The earlier one is not discarded -- it is compared against the later one, and a disagreement between the two is a contradiction rather than a supersession.

THE TWO WAYS A REGISTER CONTRADICTS ITSELF
  - TWO RETURNS THAT DISAGREE. The same person's earlier return declares something the later one does not, or the other way round. Filing again does not resolve it; somebody has to decide which is true.
  - A DECLARED RELATIONSHIP THE REGISTER RECORDS AS DISPOSED. The latest return declares a relationship, and the register's own holdings section records that relationship as disposed of on a date BEFORE the return was filed. The register contradicts itself. ⚑ THE ORDER OF THE TWO DATES IS THE WHOLE TEST: a relationship disposed of AFTER the return was filed is not a contradiction at all -- the return was correct on the day it was filed, which is the only day a return can be correct about.

THE SIX STATUSES
  satisfied, missing, stale, contradicted, not_required, not_determinable

WHAT PUTS SOMEBODY ON THE WORKLIST
  missing, stale, contradicted -- The three statuses that put a person on somebody's list today. `not_required` and `satisfied` are the two answers that mean nothing needs doing, and `not_determinable` means the register needs opening rather than the person needing chasing.

Extract these fields:
REGISTER-LEVEL FIELDS -- one value each, for the whole register
- engagement_ref (string) -- the engagement reference, verbatim from the Engagement section
- as_at_date (date) -- the date this register was assembled, from the Register As At section, as yyyy-mm-dd
- rulebook_id (string) -- the identifier of the cycle rulebook this register is kept under, verbatim -- the short code only, not the sentence after it
- register_note (string) -- the engagement administrator's own free-text remark from the Register Notes section, copied verbatim. It is a field to copy and it is NOT evidence about anybody's status

PER-ATTESTER FIELDS -- one object per person listed in Attesters On Record
- person_ref (string) -- the person reference as the Attesters On Record section states it, e.g. P-1234. Return one object per person listed there, and no others
- role (enum) one of: engagement_lead, quality_reviewer, senior_manager, specialist_advisor, engagement_administrator -- the role recorded beside this person in the Attesters On Record section, verbatim
- roster_event (enum) one of: none, joined_mid_cycle, role_vacated -- what the Roster Changes section records about THIS person: 'role_vacated' where it records the role as vacated, 'joined_mid_cycle' where it records them as having joined inside the new-joiner window, and 'none' where that section says nothing about them. Read this from the Roster Changes section only -- the roster line above cannot tell you
- cycle_opened_on (date) -- the date this person's cycle opened, from the Attesters On Record section, as yyyy-mm-dd. Where the line reads 'cycle opened -- not recorded on this register', return null. That line means the same thing on every person who carries it, whatever Roster Changes says about them
- return_filed_on (date) -- the filing date of this person's LATEST return in the Returns Filed section, as yyyy-mm-dd. That section is not grouped by person and not in date order, so read every line for this person and take the LATER filing date. Return null where the section carries no line for them at all
- return_covers_to (date) -- the end of the period the LATEST return covers, as yyyy-mm-dd. Where the line reads 'covering the period to -- not stated on the filed return', return null. Return null where there is no return at all
- declared_relationship (string) -- what the LATEST return declares, copied verbatim after 'declared:'. That includes 'no interests to report', which is a declaration and not a null. Return null only where there is no return at all
- earlier_declared_relationship (string) -- where this person has MORE THAN ONE line in Returns Filed, what the EARLIER one declares, copied verbatim. Return null where they have only one return, or none
- relationship_disposed_on (date) -- from the Holdings And Relationships On File section, the disposal date recorded against the relationship this person's LATEST return declares, as yyyy-mm-dd. Return null where that section records it as current with no disposal recorded, or carries no line for this person
- due_on (derived) -- the date this person's attestation was due. NO REGISTER STATES IT -- derive it as cycle_opened_on plus the cycle length the rulebook gives for their role (engagement_lead and quality_reviewer 180 days, senior_manager and specialist_advisor 365 days), as yyyy-mm-dd. Return null where cycle_opened_on is null, and null where the rulebook gives no cycle length for the role
- status (enum) one of: satisfied, missing, stale, contradicted, not_required, not_determinable -- this person's attestation status under the shipped rulebook. Work through five gates IN ORDER and stop at the first that fires. (1) NOT REQUIRED: roster_event is role_vacated or joined_mid_cycle, or the role is engagement_administrator, which the rulebook puts no requirement on. (2) NOT DETERMINABLE: cycle_opened_on is null so no due date can be derived, or a return is on file and return_covers_to is null so the coverage test cannot be run. (3) MISSING: no return on file at all. (4) CONTRADICTED: earlier_declared_relationship is present and differs from declared_relationship, or relationship_disposed_on is earlier than return_filed_on. (5) STALE: return_filed_on is later than due_on plus 14 days of grace, or return_covers_to is earlier than due_on. Anything that survives all five is SATISFIED. Never decide it from the administrator's note

Return a JSON object of exactly this shape:
{"engagement_ref": ..., "as_at_date": ..., "rulebook_id": ..., "register_note": ..., "attesters": [{"person_ref": ..., "role": ..., "roster_event": ..., "cycle_opened_on": ..., "return_filed_on": ..., "return_covers_to": ..., "declared_relationship": ..., "earlier_declared_relationship": ..., "relationship_disposed_on": ..., "due_on": ..., "status": ...}, ...]}
Use null for any field the register does not state.

ATTESTATION REGISTER
--------------------
Engagement
----------
ENG-1642-PV

Register As At
--------------
2026-05-31

Cycle Rulebook
--------------
IAR-1 -- illustrative, shipped with this kit; reproduces no real professional standard

Attesters On Record
-------------------
P-2421   quality_reviewer         cycle opened 2025-10-09
P-3190   engagement_lead          cycle opened 2025-08-04
P-3501   specialist_advisor       cycle opened 2025-03-19
P-2273   specialist_advisor       cycle opened 2025-04-15
P-4719   senior_manager           cycle opened 2025-05-10
P-6891   specialist_advisor       cycle opened 2025-04-08
P-4651   quality_reviewer         cycle opened 2025-08-24

Returns Filed
-------------
P-3501   filed 2026-01-25   covering the period to 2026-04-29   declared: no interests to report
P-2273   filed 2026-03-19   covering the period to 2026-02-06   declared: no interests to report
P-4719   filed 2026-03-16   covering the period to 2026-07-02   declared: no interests to report
P-6891   filed 2026-02-25   covering the period to 2026-04-23   declared: no interests to report
P-4651   filed 2026-02-17   covering the period to 2026-02-28   declared: no interests to report
P-2421   filed 2026-02-24   covering the period to -- not stated on the filed return   declared: no interests to report
P-4651   filed 2026-01-05   covering the period to 2025-08-16   declared: equity holding in a listed distribution group

Holdings And Relationships On File
----------------------------------
no holdings or relationships recorded on this register

Roster Changes
--------------
no roster changes recorded in this cycle

Register Notes
--------------
Standard rollover, nothing unusual about this engagement's declarations at all.

Raw responseThe raw response

{"engagement_ref":"ENG-1642-PV","as_at_date":"2026-05-31","rulebook_id":"IAR-1","register_note":"Standard rollover, nothing unusual about this engagement's declarations at all.","attesters":[{"person_ref":"P-2421","role":"quality_reviewer","roster_event":"none","cycle_opened_on":"2025-10-09","return_filed_on":"2026-02-24","return_covers_to":null,"declared_relationship":"no interests to report","earlier_declared_relationship":null,"relationship_disposed_on":null,"due_on":"2026-04-07","status":"not_determinable"},{"person_ref":"P-3190","role":"engagement_lead","roster_event":"none","cycle_opened_on":"2025-08-04","return_filed_on":null,"return_covers_to":null,"declared_relationship":null,"earlier_declared_relationship":null,"relationship_disposed_on":null,"due_on":"2026-01-31","status":"missing"},{"person_ref":"P-3501","role":"specialist_advisor","roster_event":"none","cycle_opened_on":"2025-03-19","return_filed_on":"2026-01-25","return_covers_to":"2026-04-29","declared_relationship":"no interests to report","earlier_declared_relationship":null,"relationship_disposed_on":null,"due_on":"2026-03-19","status":"satisfied"},{"person_ref":"P-2273","role":"specialist_advisor","roster_event":"none","cycle_opened_on":"2025-04-15","return_filed_on":"2026-03-19","return_covers_to":"2026-02-06","declared_relationship":"no interests to report","earlier_declared_relationship":null,"relationship_disposed_on":null,"due_on":"2026-04-15","status":"stale"},{"person_ref":"P-4719","role":"senior_manager","roster_event":"none","cycle_opened_on":"2025-05-10","return_filed_on":"2026-03-16","return_covers_to":"2026-07-02","declared_relationship":"no interests to report","earlier_declared_relationship":null,"relationship_disposed_on":null,"due_on":"2026-05-10","status":"satisfied"},{"person_ref":"P-6891","role":"specialist_advisor","roster_event":"none","cycle_opened_on":"2025-04-08","return_filed_on":"2026-02-25","return_covers_to":"2026-04-23","declared_relationship":"no interests to report","earlier_declared_relationship":null,"relationship_disposed_on":null,"due_on":"2026-04-08","status":"satisfied"},{"person_ref":"P-4651","role":"quality_reviewer","roster_event":"none","cycle_opened_on":"2025-08-24","return_filed_on":"2026-02-17","return_covers_to":"2026-02-28","declared_relationship":"no interests to report","earlier_declared_relationship":"equity holding in a listed distribution group","relationship_disposed_on":null,"due_on":"2026-02-20","status":"contradicted"}]}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check an engagement team's independence attestations for who to chase — 50 attestation registers. One model answered, and every answer was then graded Four different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersFour ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The box-tick floor (evals/baseline.py) and the scorer (evals/judge.py) are both pure code and cost $0.00 to run against any result set. The figure above is the ONE scored run's own token counts (50 registers, 49 of which answered) priced at Google Gemini 3 Flash's published rate — the same basis cost_per_query_usd uses. ⚠︎ IT IS NOT WHAT BUILDING THIS KIT COST. 68 live calls were made in total: the 50 above, a 3-register calibration that MEASURED max_tokens ($0.0455), 12 prompt-split calls at max_tokens=1 across three registers ($0.0128), one worked example ($0.0196), and two un-metered UI calls (one smoke check, one live screenshot) whose token counts were never recorded. The 67 metered calls together project to $0.9611 on this card.

Cost driversWhat actually moves the bill

• OUTPUT, NOT INPUT, AND THAT IS THE SHAPE OF A ROSTER REPLY. 5,314 output tokens per register on average against 3,178 input — output is 75 pct of the bill. Every sibling extraction kit in this series is the other way round, because they return one record and this returns a list of them.
• Provider-side reasoning. 229,975 of r001's 260,390 output tokens (88.3 pct) were reasoning rather than the JSON register, on a reply that is about 600 tokens of actual content. Reasoning was left at the provider's default and nothing here has measured what turning it off would cost in accuracy — or whether it would have saved the register the ceiling ate.
• The fixed prefix — system prompt (1,002 tokens), attestation rulebook (566) and field schema (1,169) — is 2,737 of 3,226 tokens on the worked example, 85 pct, and is byte-identical on every one of the 50 calls. The register itself is 489 tokens. THE SCHEMA IS THE LARGEST SINGLE PART, not the rulebook: eleven per-person fields each need a hint long enough to be unambiguous.

Your volumeWhat it costs at your volume

Linear in registers: each call is independent, carries the same fixed prefix and shares nothing with its neighbours, so 500 registers cost ten times 50. Nothing amortises — there is no index and no cache. ⚠︎ AND IT DOES NOT SCALE IN ROSTER SIZE AT ALL, WHICH IS THE LIMIT THAT BITES FIRST. The reply is one object per person, so a register of 40 people is roughly six times the output of one of 7 — and one seven-person register already exceeded a 16,000-token ceiling on this run. At that size the roster has to be chunked across calls, which is a different kit and not a bigger prompt.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 attestation registers, one completion call each, on the fast tier. 49 answered; the 50th hit the output ceiling and returned nothing, and its call is not in the token totals below because the provider reported no usage for a reply that never completed — so these figures cover 49 calls, not 50.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the attestation register into addressable sections, pure code — 50 registers into 450 sections in 0.007 seconds

# Cut a document into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections carry each field, pure code — roster_event is mapped to Roster Changes ONLY, because the roster line cannot tell a mid-cycle joiner from a record nobody can read; status is mapped to the four sections the rule actually reads and never to Register Notes; and the Prepared By section is mapped by nothing and never sent

You change it to: map fields to your own register's headings; unmatched falls back to the whole document

# Pick which sections plausibly carry each field. Pure code -- the last deterministic step before

SECTION_HINTS = {
def for_field(secs, field):
def plan(secs, fields):

src/rulebook.pyrulebook

the five-gate status rule and the due-date arithmetic, loaded from data/rulebook.json — roles that carry a requirement, cycle length by role, the grace window, the coverage test and the two ways a register contradicts itself. Data, not code, so a forker can open it and disagree with it

# THE ATTESTATION RULEBOOK, loaded from data/rulebook.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
STATUSES = ("satisfied", "missing", "stale", "contradicted", "not_required", "not_determinable")
WORKLIST = ("missing", "stale", "contradicted")
ROSTER_EVENTS = ("none", "joined_mid_cycle", "role_vacated")
def load(path=RULEBOOK_PATH):
RB = load()
ROLES_REQUIRING = tuple(RB["roles_requiring_attestation"])
ROLES_NOT_REQUIRING = tuple(RB["roles_not_requiring_attestation"])

src/prompt.pyprompt

assemble one call for the whole register — four register-level fields and eleven per person — with the rulebook RENDERED into it from data/rulebook.json rather than restated in prose, so the instructions and the gold labels cannot drift about the same arithmetic

# Assemble the register-monitoring prompt. ONE prompt per register, every person on it in one

SYSTEM = (
def rulebook_block(rb=None):
RULEBOOK_TEXT = rulebook_block()
def field_schema(fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key — plus the pure-code status rule run over every person the reply returned, and the register-level routing flag downstream of it

You change it to: the routing rule itself — this kit ships one predicate over the computed statuses (any contradicted, or any not determinable) and a real practice weighs who the person is, how close sign-off is and who already knows. It is deliberately NOT the same function as the status rule, so changing WHERE A REGISTER GOES does not change WHAT A STATUS MEANS

# Extract one attestation register: segment, select, prompt, ONE model call, then the pure-code

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 16000
def load_fields():
def attester_key(fields):
def scored_attester_fields(fields):
def load_doc(register_id):
def documents():

evals/judge.pyjudge

score four things separately and never fold them together: field extraction, the pure-code status verdict, the model's own date arithmetic, and the FALSE-ALARM RATE on the worklist — plus the not-determinable reach and the routing flag, each with its own confusion matrix, all pure code

# Score an attestation-register run. PURE CODE -- gold is exact and every answer is one value, so

def norm(v):
def equal(field, got, want):
def _rows_by_ref(attesters, key="person_ref"):
def score(fields, records, golds):
def _matrix(rows, positive):
NOTHING_TO_DO = ("satisfied", "not_required")
def score_statuses(fields, records, golds):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3177 input and 5314 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's attestation registers are entirely synthetic (tools/build_corpus.py, seed 20260822): no real firm, standard-setter, regulator, client, engagement or person exists in the corpus, every person reference and relationship description is invented, and nothing was fetched from anywhere. The only outbound traffic the kit makes is one chat-completion request per register, carrying the mapped sections of one register plus the fixed prompt and the shipped rulebook. Nothing is written outside the kit directory, there is no database, no scheduler, no queue, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser — which matters here because this kit deliberately surfaces transport errors that carry a URL.

The experimentWe did not attack it — and four of five boundaries hold

The four boundaries that hold were confirmed by reading the code, not by a run: no code path chases, files, signs off or clears; nothing in the kit schedules, polls, subscribes or stores anything, so there is no unattended path at all; the status rule and the routing rule read only named values and a list of statuses, so the administrator's note cannot reach either; and the key and base URL are redacted out of any error before it is returned. The fifth is open. An indirect prompt injection needs a field an outside party authored, and this kit has exactly one — the administrator's note — which makes it the obvious place to attack and the reason the gap is named rather than glossed. Confirmed by reading the code, not by a run, on 2026-08-22 — no attack run was fired; see redteam.why.

Four boundaries hold, confirmed by reading the code rather than by an attack run. The fifth is open and is named as open — an indirect prompt injection in the administrator's note is exactly the attack this kit's shape invites, and nothing here has tried it.

The result0 attack trials, and four of five boundaries checked here hold on evidence. The one that does not is the one this kit's shape invites: the administrator's note is free text somebody other than the attester wrote, and whether an instruction hidden in it could move a status is unmeasured.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted ambiguity is a REGISTER mismatch — a settled-sounding note over a register with four people to chase — which measures whether a model runs the rulebook when the prose points the other way. It does not measure whether a model obeys an instruction hidden in the same field. Those are different failures and only one of them is measured here.

HonestyWhat this does not prove

• Whether an indirect prompt injection in the register_note field could move a status. No attack run exists.
• Whether the kit behaves safely against a hostile provider — a response body crafted to break the JSON extraction in src/prompt.py::parse, or to return an enormous roster. parse() fails closed to an empty register and an empty list, which the harness records as a failed document, and nothing beyond that was tested.
• Whether a real firm's attestation paperwork would carry anything sensitive this kit mishandles. The corpus has no personal data by construction; a real register carries named people, their family and financial relationships and a client's identity, and none of that path is exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

Two rules, in two places, and keeping them apart is the point. PER PERSON: src/rulebook.py::decide() walks five gates in order and stops at the first that fires — is this person required to attest at all (a vacated role, a mid-cycle joiner or a role the rulebook puts no requirement on is `not_required`); can the register be read on them at all (no cycle date, or a return whose covered period is not stated, is `not_determinable`); is there a return (`missing`); does the register contradict itself about them (two returns that disagree, or a relationship disposed of BEFORE the return that declares it — `contradicted`); was it in time and about the right window (`stale`). Anything surviving all five is `satisfied`. PER REGISTER: src/extract.py::compute() raises needs_owner_review when the register carries at least one `contradicted` person or at least one `not_determinable` one. Both run afterwards in pure code, over whatever the model returned, and never over gold. A reply missing a value returns None rather than a status — an unknown is not a pass.

src/rulebook.py::decide(), called by src/extract.py::extract() on every person of every register, by the local app on every click, and by evals/baseline.py's judge path so a box-tick floor and a real run are routed by identical code. The register-level routing rule lives in a SEPARATE function, src/extract.py::compute(), which reads only the list of computed statuses. One definition of a status, three readers — the corpus generator wrote gold with it, src/prompt.py states it to the model in words, and evals/judge.py re-runs it over the model's own values.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a monitor. Nothing here polls, subscribes, schedules, alerts, escalates, chases, files, signs a register off or clears anybody. It reads one snapshot somebody else assembled, proposes a worklist, and stops. There is no state in the kit at all.
• It is NOT a check on whether the extracted values are right. If the model misreads a date and then the rule reasons correctly from the misreading, the person is flagged — or not flagged — on a wrong fact and this rule cannot tell. The consistency diagnostic in evals/judge.py is the closest thing to that check, it is reported separately, and it is blind to exactly the same case.
• It is NOT anybody's independence policy. The rulebook was written for this kit: no real professional standard, standard-setter's rule, regulator's requirement or firm policy was consulted, and none is reproduced. No real firm, standard-setter, regulator, standard number, client or person is named anywhere in the kit.
• It is NOT a judgement about the rulebook. The routing flag reads the statuses the rulebook produced; if the rulebook is wrong, the flag is confidently wrong with it.
• It has NO notion of “not due yet”. Every required attester in this corpus is past their due date by construction, and evals/check_labels.py asserts it. On a real register that case is common, and this kit would call every one of those people missing — which would be the largest source of false alarms a forker meets on day one.

WatchedWhat is watched, and why that one

The kit measures 13 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 6 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 50, extraction_cells 3000, failures 0, refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

extraction · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on documents, extraction_cells, failures, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

extraction · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 7 chips that all say so.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the evidence on any change to src/rulebook.py or data/rulebook.json (which moves every status, every derived date and gold at once), on any change to compute() in src/extract.py, on any change to data/fields.json's allowed values for status or roster_event, on any change to MAX_TOKENS, and on any corpus regeneration. Changing WHERE A REGISTER GOES is a policy change and must be re-scored, even though it never changes what a status means.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the rulebook is a JSON file read with json, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install. ⚑ AND THERE IS DELIBERATELY NO SCHEDULER, NO QUEUE AND NO STORE — the obvious reach for anything called a monitor, and the wrong one here. A kit is one call per document with a checkable gold; adding a poller would add state nobody can score and would not make a single status more correct.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per register — segment, select, prompt, one call, parse, compute per person, route the register — with no branch, no loop and no agent. The only fan-out is a list comprehension over the roster the reply returned. Anything that looks like orchestration in a kit this size is a diagram of a straight line.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain: the JSON extraction from a fenced reply, the roster alignment by person reference, the backoff policy, the section regex and the .env reader are all code somebody has to own. The list alignment in particular is code a framework would have given you — and getting it wrong silently drops people off a monitoring roster, which is why roster coverage is published as its own number.
• No framework means no framework's ecosystem — no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. A JSON-schema-constrained decode would plausibly have prevented the one register this run lost, and nothing here tested that.
• The provider abstraction covers exactly two wire formats. A third provider is one function and one dict entry, and until somebody writes it the kit runs on two shapes.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-attestation-watch on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b000-attestation-watch-boxtick, t000-attestation-watch-stub recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Every engagement reference, person reference, role, relationship description and administrator note is invented; no real firm, standard-setter, regulator, standard number, client or person is named anywhere in this kit. The cycle rulebook it ships (data/rulebook.json) is ILLUSTRATIVE and is not an authority. It was written for this kit and reproduces no real professional standard, standard-setter's rule, regulator's requirement or firm policy. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-field exact match, light normalisation

Does the reply's value for each field match gold, after trimming whitespace and punctuation? Rows are aligned by person_ref, and THAT KEY IS NOT ITSELF A SCORED CELL — scoring the field a row was matched by would make every joined row a free hit, so roster coverage is reported separately. Seven of the eleven per-person fields are legitimately nullable, and each null is a stated fact rather than a convenience: an unrecorded cycle, no return on file, a covered period the register does not state, no relationship declared, no earlier return, no disposal recorded, and a role for which no due date can be derived at all.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every call, and the same field-match logic the free floor is scored by — a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same values the register states; evals/check_labels.py asserts every non-nullable field is populated on every row, that no person reference is duplicated within a register, that every gold status and every gold due date is the shipped rule's own output, and that each hard case clears its own floor, before any run is allowed to spend.

How tight can the band be? There is no tolerance band — it is exact match after trimming whitespace and punctuation, never a continuous score to round.

Cadence: Re-score on any change to data/rulebook.json, src/prompt.py, data/fields.json or tools/build_corpus.py — the first three change what is asked, the fourth changes what is asked about. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe status verdict, six-way and collapsed onto the worklist

Two questions at once. Six-way: did the PURE-CODE rule, run over the values the model returned, land on the same status the rulebook produces — satisfied, missing, stale, contradicted, not_required or not_determinable? And the one a monitoring queue is actually made of: OF EVERY PERSON THIS KIT PUT ON THE WORKLIST, HOW MANY DID NOT BELONG THERE? A false alarm is counted STRICTLY — a flagged row whose gold status is satisfied or not_required, the two answers that mean nothing needed doing. A flagged not_determinable row is counted separately as MISROUTED, because somebody does have to open that file; they have to open it for a different reason. The other direction, a person who needed action and was told they did not, is counted and named on its own as missed_breach.

MethodHow the test was run

evals/judge.py::score_statuses, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's status is re-derived inside the grader by the same rule the kit publishes — src/rulebook.py::decide(), reading data/rulebook.json — so the truth this grades against can never be a separately-typed label that drifted from the rule.

How tight can the band be? No threshold — the status is one of six allowed values, and a person the reply never returned is counted as unanswered rather than folded into a correct cell. On a monitoring queue that distinction is the whole difference between a person nobody chased and a person somebody cleared.

Cadence: Re-run whenever data/rulebook.json or src/rulebook.py changes, whenever the corpus is regenerated, and on any provider or model change.

The decisionWhen to reach for it

In one lineThe derived due date

Did the MODEL's own due_on match the derivation — the cycle-opened date plus the rulebook's cycle length for the role? NO REGISTER STATES A DUE DATE, so this is arithmetic rather than copying. It is scored on its own because an obligation found but mis-dated is a different failure from one missed: the first sends somebody to the wrong week, the second sends nobody at all. ⚑ THE MODEL'S NUMBER DECIDES NOTHING — the kit publishes the code's date — and that is exactly what makes this a clean measurement.

MethodHow the test was run

evals/judge.py::score_statuses, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/rulebook.py::due_on(), the same function the kit publishes and the same one tools/build_corpus.py wrote gold with.

How tight can the band be? No threshold — a date matches or it does not, compared as a string after trimming.

Cadence: Re-run whenever data/rulebook.json's cycle lengths or role list change.

The decisionWhen to reach for it

In one lineneeds_owner_review against the same rule run over gold

Does the pure-code routing decision — this register carries at least one contradicted person or at least one not_determinable one — land on the same registers it would land on if every status had been computed from perfectly-read values? It is a BUSINESS CONDITION, so unlike a self-consistency check it genuinely needs labels, and saying so is half of what makes the number believable.

MethodHow the test was run

evals/judge.py::score_statuses, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute(), the same function the run uses, applied to the statuses derived from GOLD's own values. One rule, one input, so a change to the rule moves both sides together and the grader cannot silently grade an old policy.

How tight can the band be? No threshold — one predicate over a list of statuses.

Cadence: Re-run whenever compute() changes. Changing WHERE A REGISTER GOES is a policy change and must be re-scored, even though it never changes what a status means.

The decisionWhen to reach for it