The business caseThe problem this solves

A site's permit obligations are recorded in a register: what each condition requires, how often, what has been done and when. Somebody has to turn that into a worklist — which obligations need action now, by when, and which ones cannot be determined from what is written down. The register works against them in four specific ways: it keeps conditions that have been superseded or waived, it records a filing date beside a reporting period that disagrees with it, it lets an entry be logged with no date at all, and it carries the site's OWN status flag, which is written by the party being checked and is wrong on 40 pct of the rows here. Somebody opening each site's obligation register, going down it condition by condition, remembering which cycle each obligation runs on and how much notice its type needs, working out the next due date from whatever the record says was last done, and deciding which rows go on this week's list. It is a few minutes per site and it is every site, every cycle, and the part that goes wrong is not the arithmetic: it is reading a filing DATE where the reporting PERIOD is the answer, computing a due date on a condition that was superseded two amendments ago, and treating a line with the date column empty as though it had been done.

The inputThe actual obligation registers

The corpus is 50 obligation registers, 0.11 MB (txt 50). Plain text, one format, invented rather than fetched — a real obligation register states a real site's actual compliance position under a real permit, which is commercially sensitive on one side and legally consequential on the other, and there is no public corpus of (register, correct worklist) pairs for the same reason there is no public corpus of bank statements. Generating it also makes the label mechanical: gold's status and due date are the rulebook lookup over the same values the register states, never somebody's reading of the site's own flag.

Site
----
Northreach Ridge Operation (SITE-NR-9441)

Permit
------
MP-9700-J, issued by the Ninth District Minerals and Environment Office

Register Date
-------------
2026-01-17

Register Note
-------------
Site self-assessment at this review: compliance position considered satisfactory this quarter.

Condition C-9.3
---------------
Requirement: annual rehabilitation progress report for the calendar reporting year
Obligation type: periodic_report
Condition state: active
Last recorded as done: 2025-11-04
Period credited: 2023 reporting year
Stated due date: not stated
Trigger event: not applicable to this condition
Register flag: attention

Condition C-11.9
----------------
Requirement: quarterly boundary noise reading at station NS-15
Obligation type: monitoring_reading
Condition state: active
Last recorded as done: 2025-12-05
Period credited: not applicable to this condition
Stated due date: not stated
Trigger event: not applicable to this condition
Register flag: closed

Condition C-8.1
---------------
Requirement: annual renewal of the pollution incident financial provision
Obligation type: financial_assurance
Condition state: active
Last recorded as done: 2024-11-03
Period credited: not applicable to this condition
Stated due date: not stated
Trigger event: not applicable to this condition
Register flag: attention

Condition C-3.6
---------------
Requirement: annual water balance report for the calendar reporting year
Obligation type: periodic_report
Condition state: active
Last recorded as done: 2025-11-04
Period credited: 2023 reporting year
Stated due date: not stated
Trigger event: not applicable to this condition
Register flag: on track

The outcomeWhat a good result looks like

A row per condition with its status, the date it falls due and the rulebook's own reason for that status — plus a proposed worklist of the obligations needing action, the ones the register cannot answer named as such rather than cleared, and one pure-code escalation: this register carries something already overdue that its own flag calls fine. Nothing here files, submits, notifies, renews or clears, and nothing continues after the call returns.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Turning a book of site obligation registers into this week's worklist before a compliance or approvals desk opens them — the fast tier — one call per register, 6.6s at the median, $0.0042532 a register
  0 false alarms of the 126 obligations it raised and 0 missed of the 126 that needed action, against a free register-flag floor that raises 25 rows needing nothing and misses 53 that do. It also derived all 172 due dates exactly, which the floor cannot do at all.
• Deciding whether the rulebook block belongs in the prompt at all — take it out
  Measured, not argued. It is 458 of the worked example's 2,223 input tokens (20.6 pct) and the model is explicitly told it will not compute a status. Run a001 removed it, kept the system prompt and every field hint, and re-fired the 17 registers where it could plausibly have mattered: 763 of 763 cells, 89 of 89 statuses, 65 of 65 due dates, all 22 wrong-period decoy rows still correct, on 21.0 pct fewer input tokens.
• Deciding whether this shape is worth buying at all, from these numbers — not from these numbers alone — run it on your own registers first
  What the numbers DO support is narrow and real: taking a compliance status from a register's own flag is measurably unsafe on this shape of decision, in both directions at once; and a monitor that can say 'cannot determine' catches 26 of 26 registers' worth of unanswerable rows rather than clearing them.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-permit-obligations. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts on 127.0.0.1:8853, the 50-register picker populates, the register panel draws its three empty rows, the worklist says “Nothing has been read yet” in words rather than rendering an empty table, and the shipped rulebook renders in full from data/rulebook.json. Clicking Read register returns the no-key sentence rather than a stack trace, and the escalation row reads “not computed — one of the values the rule needs was missing” rather than “no”. python -m evals.check_labels and python -m evals.run --baseline both pass with no network access at all.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per obligation register

Current processWhat it replaces

Somebody opening each site's obligation register, going down it condition by condition, remembering which cycle each obligation runs on and how much notice its type needs, working out the next due date from whatever the record says was last done, and deciding which rows go on this week's list. It is a few minutes per site and it is every site, every cycle, and the part that goes wrong is not the arithmetic: it is reading a filing DATE where the reporting PERIOD is the answer, computing a due date on a condition that was superseded two amendments ago, and treating a line with the date column empty as though it had been done.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 2 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings; every permit condition is its own headed block, and a register with no headings falls back to one whole-document segment

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 50 registers cut into 468 sections in three thousandths of a second, in process, with no model and no network.

LicenceLicence

MIT — this repository's own licence. Every site name, site and permit identifier, administering office, condition identifier and waiver reference is invented, and no real operator, mine, site, permit, regulator or regulation is named. The obligation rulebook this kit ships (data/rulebook.json) is ILLUSTRATIVE and is not an authority. It was written for this kit and reproduces no permit, licence condition, regulator's guidance, industry code or statutory schedule.

Bring your ownBring your own obligation registers

REPLACE data/rulebook.json FIRST. It is this kit's own construction, it resembles no real permit, and everything downstream reads it: src/rulebook.py loads it, src/prompt.py renders it into every call, and tools/build_corpus.py writes gold with it. Then replace data/corpus/*.txt, write your own data/fields.json (both halves — register-level and per-condition), and supply a gold record per register. SECTION_HINTS and SECTION_PREFIXES in src/select.py map fields to section headings and will need editing for a different register layout; when they do not match, selection falls back to the whole document — slower, more expensive, always correct. compute() in src/extract.py is this kit's own escalation rule and is the second thing to replace.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 2,223 tokens assembled, 1,052 are contexts — 47% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on REG-0003 — the same register the worked example above was captured on, so the four parts sum to exactly the 2223 input tokens that call actually reported. Four calls at max_tokens=1, each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. kits/UC0049-permit-obligations/results/tokens-p001-permit-obligations.json.

Full promptThe prompt in full

You read a SITE PERMIT OBLIGATION REGISTER and report what it records. You return JSON and nothing else.

YOU DO NOT DECIDE WHETHER ANYTHING IS OVERDUE, DUE SOON, OR CLEAR. A separate piece of code works that out from the values you report, against the rulebook shown below. Your job is to report the register faithfully, including the places where it records nothing.

RULES, in order of importance:
1. If the register does not state a field, return null for it. Do not infer it, do not carry a value across from another condition block, and do not use what you know about the world.
2. RETURN ONE ENTRY PER CONDITION BLOCK, in the order the register prints them, including blocks that are superseded, waived, or plainly closed out. A register accretes; the rows nobody deleted are exactly the rows this reading is about.
3. `condition_state` IS THE FIRST WORD OF THE 'Condition state' LINE AND NOTHING ELSE. A superseded or waived condition still carries dates, and those dates do not change the state. Report 'superseded' or 'waived' even where the rest of the block looks like ordinary work that is running late.
4. `last_done` IS A DATE OR IT IS NULL. Where the line says the entry was logged with no date recorded, return null -- not today's date, not the amendment date further up the block, and not a date from another condition. Where the line says it does not apply to this condition, return null.
5. FOR AN ANNUAL REPORT, `period_credited` IS THE REPORTING YEAR THE FILING WAS CREDITED TO, taken from the 'Period credited' line and never from the filing date above it. The two routinely disagree and the period is the one that counts.
6. `trigger_state` DISTINGUISHES 'not_occurred' FROM 'not_recorded'. The first means the register says the event has NOT happened. The second means the register does not say either way. They are different answers.
7. `register_flag` is the site's own self-assessment of the row. Copy it verbatim. It is frequently wrong and it is never evidence about anything.
8. Copy identifiers verbatim. Write every date as YYYY-MM-DD, exactly as the register does.
9. Use the exact allowed value for a field that lists them, and return every field named in the schema on every entry, even when the answer is null.

HOW THE STATUS IS WORKED OUT AFTERWARDS (context for you; you do not compute it). This is an ILLUSTRATIVE rulebook shipped with this kit, not a real permit.

OBLIGATION TYPES
  event_triggered:       nothing is due until the trigger event occurs; then the stated due date governs; action window 30 days
  financial_assurance:   due 365 days after the last recorded date; action window 60 days
  inspection:            due 180 days after the last recorded date; action window 30 days
  monitoring_reading:    due 90 days after the last recorded date; action window 30 days
  periodic_report:       covers a calendar year; due 03-31 of the year after the year it covers; action window 30 days

CONDITION STATES -- read before any date
  active:        the condition binds and its status is computed.
  superseded:    the condition has been replaced by a later one in a permit amendment. It is still printed on the register -- registers accrete -- and it binds nothing. Status is `not_binding` whatever its dates say.
  waived:        the administering body has waived the condition in writing, with a reference. It binds nothing. Status is `not_binding` whatever its dates say.

TRIGGER STATES
  occurred:        the register records the trigger event as having happened, and states the resulting due date.
  not_occurred:    the register records that the trigger event has not happened. Nothing is due -- `not_yet_due`, and it is not an omission.
  not_recorded:    the register does not say either way. `not_determinable` -- an unrecorded trigger is not the same fact as a trigger that did not fire.
  not_applicable:  the row is not an event-triggered condition.

THE STATUS IS ONE OF: overdue, due_in_window, not_yet_due, not_binding, not_determinable
`not_determinable` is a real answer. The register not carrying what the rule needs is a fact to report, not a gap to fill in.

Extract these register-level fields:
- site_id (string) -- the site identifier printed in brackets after the site name in the Site section, verbatim
- permit_no (string) -- the permit number in the Permit section, verbatim and without the issuing office
- register_date (date) -- the date this register was drawn, from the Register Date section, as YYYY-MM-DD. Every status on this register is measured against this date

Then return `obligations`: a list with ONE ENTRY PER CONDITION BLOCK on the register, in the order printed, each entry carrying exactly these fields:
- condition_id (string) -- the condition identifier from the block heading, e.g. C-7.3, verbatim
- obligation_type (enum) one of: monitoring_reading, inspection, financial_assurance, periodic_report, event_triggered -- the obligation type stated on the block, verbatim
- condition_state (enum) one of: active, superseded, waived -- the FIRST WORD of the Condition state line and nothing else. A superseded or waived condition is still printed on the register and still carries dates; report the state exactly as it stands and do not let its dates change your answer
- last_done (date) -- the date on the 'Last recorded as done' line, as YYYY-MM-DD. Return null when the line says the entry was logged with no date recorded, and null when it says the line does not apply to this condition. Do not guess a date from anything else on the block
- period_credited (string) -- for an annual report, the four-digit reporting YEAR the last filing was credited to, e.g. '2024' -- taken from the 'Period credited' line and NOT from the filing date above it. Return null where the line says it does not apply
- stated_due (date) -- the date on the 'Stated due date' line, as YYYY-MM-DD. Return null where the line says no date is stated
- trigger_state (enum) one of: occurred, not_occurred, not_recorded, not_applicable -- what the Trigger event line records. 'not_occurred' means the register says the event has NOT happened, which is a recorded fact; 'not_recorded' means the register does not say either way. They are different answers and must not be merged
- register_flag (enum) one of: on track, attention, closed -- the site's own Register flag for this condition, copied verbatim. It is the site's self-assessment, it is often wrong, and it is NEVER an input to anything -- report it as stated

Return a JSON object with exactly these keys: site_id, permit_no, register_date, obligations
Use null for any field the register does not state.

PERMIT OBLIGATION REGISTER
--------------------------
Site
----
Kestrel Hollow Operation (SITE-KH-3501)

Permit
------
MP-3516-C, issued by the Ninth District Minerals and Environment Office

Register Date
-------------
2026-01-24

Condition C-3.7
---------------
Requirement: half-yearly geotechnical inspection of the waste rock dump
Obligation type: inspection
Condition state: active
Last recorded as done: entry logged, date not recorded
Period credited: not applicable to this condition
Stated due date: not stated
Trigger event: not applicable to this condition
Register flag: attention

Condition C-9.5
---------------
Requirement: quarterly surface water sampling at discharge point DP-21
Obligation type: monitoring_reading
Condition state: superseded - replaced by condition C-21.1 in the permit amendment of 2024-12-24
Last recorded as done: 2025-04-28
Period credited: not applicable to this condition
Stated due date: not stated
Trigger event: not applicable to this condition
Register flag: attention

Condition C-12.8
----------------
Requirement: annual environmental performance report for the calendar reporting year
Obligation type: periodic_report
Condition state: active
Last recorded as done: 2025-11-17
Period credited: 2023 reporting year
Stated due date: not stated
Trigger event: not applicable to this condition
Register flag: on track

Condition C-6.5
---------------
Requirement: annual renewal of the rehabilitation security held against this permit
Obligation type: financial_assurance
Condition state: active
Last recorded as done: 2025-03-14
Period credited: not applicable to this condition
Stated due date: not stated
Trigger event: not applicable to this condition
Register flag: attention

Condition C-8.9
---------------
Requirement: annual renewal of the pollution incident financial provision
Obligation type: financial_assurance
Condition state: active
Last recorded as done: 2025-03-25
Period credited: not applicable to this condition
Stated due date: not stated
Trigger event: not applicable to this condition
Register flag: attention

Condition C-10.8
----------------
Requirement: half-yearly inspection of the haul road water crossings
Obligation type: inspection
Condition state: active
Last recorded as done: 2025-09-18
Period credited: not applicable to this condition
Stated due date: not stated
Trigger event: not applicable to this condition
Register flag: on track

Raw responseThe raw response

{"site_id":"SITE-KH-3501","permit_no":"MP-3516-C","register_date":"2026-01-24","obligations":[{"condition_id":"C-3.7","obligation_type":"inspection","condition_state":"active","last_done":null,"period_credited":null,"stated_due":null,"trigger_state":"not_applicable","register_flag":"attention"},{"condition_id":"C-9.5","obligation_type":"monitoring_reading","condition_state":"superseded","last_done":"2025-04-28","period_credited":null,"stated_due":null,"trigger_state":"not_applicable","register_flag":"attention"},{"condition_id":"C-12.8","obligation_type":"periodic_report","condition_state":"active","last_done":"2025-11-17","period_credited":"2023","stated_due":null,"trigger_state":"not_applicable","register_flag":"on track"},{"condition_id":"C-6.5","obligation_type":"financial_assurance","condition_state":"active","last_done":"2025-03-14","period_credited":null,"stated_due":null,"trigger_state":"not_applicable","register_flag":"attention"},{"condition_id":"C-8.9","obligation_type":"financial_assurance","condition_state":"active","last_done":"2025-03-25","period_credited":null,"stated_due":null,"trigger_state":"not_applicable","register_flag":"attention"},{"condition_id":"C-10.8","obligation_type":"inspection","condition_state":"active","last_done":"2025-09-18","period_credited":null,"stated_due":null,"trigger_state":"not_applicable","register_flag":"on track"}]}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch overdue permit obligations in a mine site's register — 50 obligation registers. One model answered, and every answer was then graded Five different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran these runs publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersFive ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The register-flag floor (evals/baseline.py) and the scorer (evals/judge.py) are both pure code and cost $0.00 to run against any result set. The figure above is the SCORED run's own token counts (50 registers) priced at Google Gemini 3 Flash's published rate — the same basis cost_per_query_usd uses. ⚠︎ IT IS NOT WHAT BUILDING THIS KIT COST. 83 live calls were made in total. Four further paid passes are committed in results/ and sit outside the figure above: a 6-register calibration that MEASURED max_tokens ($0.0242085), the 17-register prompt ablation ($0.072733), 4 prompt-split calls ($0.002779) and one worked example ($0.0037545) — $0.316137 across everything with committed token counts. Five further live calls are named rather than priced, because their token counts were not written to any committed file: three completeness probes on REG-0003 after a dropped condition block was spotted, and two UI screenshot clicks.

Cost driversWhat actually moves the bill

• The fixed prefix — system prompt (612 tokens), rulebook (458) and field schema (559) — is 1629 of 2223 tokens on the worked example, 73 pct, and is byte-identical on every one of the 50 calls. The register itself is 594 tokens.
• ⚑ A FIFTH OF THE INPUT IS A RULEBOOK THE MODEL IS TOLD IT WILL NOT APPLY — and removing it moved nothing. 458 tokens of every call, measured; run a001 stripped it and scored 763 of 763 cells and 89 of 89 statuses on the subset where it could have mattered. This is the one cost driver on this page with a measured answer attached.
• Provider-side reasoning. 30792 of r001's 52760 output tokens (58.4 pct) were reasoning rather than the JSON record. Reasoning was left at the provider's default and nothing here has measured what turning it off would cost in accuracy.
• Output length scales with the register, unlike the input. The reply is one JSON entry per condition, so a 7-condition register costs materially more to answer than a 4-condition one — and output is 74 pct of this kit's bill on the published card.

Your volumeWhat it costs at your volume

Linear in REGISTERS: each call is independent, carries the same fixed prefix and shares nothing with its neighbours, so 500 registers cost ten times 50. Nothing amortises — there is no index and no prefix cache. ⚠︎ BUT IT IS NOT LINEAR IN CONDITIONS, AND THAT IS THE ONE THAT BITES. The reply is one JSON entry per condition, so a site with 60 conditions rather than 5 is not ten times the input, it is roughly ten times the OUTPUT — which is three quarters of the bill here — and it is the same axis along which one live read already dropped a condition block. At that size the shape has to change to batched conditions with a reconciliation step, which is a different kit.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 permit obligation registers carrying 268 conditions, one completion call each, on the fast tier. The calibration, the prompt ablation, the prompt-split measurement, the worked example and the UI captures are recorded separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the register into addressable sections, pure code — one section per permit condition, which is what makes a per-condition citation possible at all

# Cut a document into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections carry each field, pure code — the obligations map to every Condition block by heading PREFIX, and the site's own summary Register Note is mapped by nothing and never leaves the machine

You change it to: map fields to your own register's headings; the prefix map is how a field says 'every block of this kind'. Unmatched falls back to the whole document

# Pick which sections plausibly carry each field. Pure code -- the last deterministic step before

SECTION_HINTS = {
SECTION_PREFIXES = {
def for_field(secs, field):
def for_condition(secs, condition_id):
def plan(secs, fields):

src/rulebook.pyrulebook

the status decision, loaded from data/rulebook.json — five ordered checks over intervals, per-type action windows, condition states and trigger states. Data, not code, so a forker can open it and disagree with it

# THE OBLIGATION RULEBOOK, loaded from data/rulebook.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
STATUSES = ("overdue", "due_in_window", "not_yet_due", "not_binding", "not_determinable")
ACTIONABLE = ("overdue", "due_in_window")
def load(path=RULEBOOK_PATH):
R = load()
TYPES = R["obligation_types"]
CONDITION_STATES = tuple(R["condition_states"])
TRIGGER_STATES = tuple(R["trigger_states"])

src/prompt.pyprompt

assemble one call for the whole register, with the rulebook RENDERED into it from data/rulebook.json rather than restated in prose — as context for why each field matters, since the model is told it will not compute the status

# Assemble the register-reading prompt. One prompt per register, every condition block in it.

SYSTEM = (
def rulebook_block(r=None):
RULEBOOK_TEXT = rulebook_block()
def field_schema(fields):
def schema_block(fields, ob_fields):
def build(doc_text, secs, fields, ob_fields, selector):
def parse(raw, fields, ob_fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key — plus the pure-code status rule over every returned row, the proposed worklist, and the business-condition escalation

You change it to: the escalation rule itself — this kit ships two values (something overdue AND the site's own flag quiet about it) and a real compliance function weighs which condition lapsed, by how long, and whether it is reportable in its own right. It is deliberately NOT the same function as the status rule, so changing WHAT GETS ESCALATED does not change WHAT IS OVERDUE

# Read one site's permit obligation register: segment, select, prompt, one model call, then the

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 8000
def load_fields():
def load_obligation_fields():
def load_doc(register_id):
def documents():
def statuses(values):

evals/judge.pyjudge

score field accuracy, the five-way status, the DERIVED DUE DATES and the FALSE-ALARM RATE separately, plus the cannot-determine and escalation matrices, all pure code

# Score a register-reading run. PURE CODE -- gold is exact and every answer is one value, so `==`

def norm(v):
def equal(field, got, want):
def _flat(gold_ob):
def _derived(gold_ob):
def score(fields, ob_fields, records, golds):
def _cell_row(reg_id, cid, f, got, want, span, spannable, by_field):
def _matrix(rows, positive):
def score_statuses(records, flags, golds):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 2175 input and 1055 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's obligation registers are entirely synthetic (tools/build_corpus.py, seed 20260822): no real site, operator, permit, administering body or condition exists in the corpus, and nothing was fetched from anywhere. Every site name, permit number, office, condition identifier and waiver reference is invented. The only outbound traffic the kit makes is one chat-completion request per register, carrying the mapped sections of that register plus the fixed prompt and the shipped rulebook. Nothing is written outside the kit directory, there is no database, no auth, no multi-tenancy and no scheduler, and the local UI binds 127.0.0.1 only.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser — which matters here because this kit deliberately surfaces transport errors, and a transport error string is the one that carries a URL.

The experimentWe did not attack it — and three of four boundaries hold

The three boundaries that hold were confirmed by reading the code, not by a run: nothing in the kit polls, schedules, files, notifies or clears, and there is no dependency that could; no code path acts on a status; and the status rule reads typed values only, so the site's own flag cannot reach it. The fourth is open. An indirect prompt injection needs a field an outside party authored, and a permit obligation register is full of them — which makes it the obvious place to attack and the reason the gap is named rather than glossed. Confirmed by reading the code, not by a run, on 2026-08-22 — no attack run was fired; see redteam.why.

The first three boundaries hold, confirmed by reading the code rather than by an attack run. The fourth is open and is named as open. Note that the first gate is not a normal one for this estate: it exists because the PATTERN'S NAME makes a claim the kit does not, and a reader who assumes a monitor monitors would be wrong about the most important thing on the page.

The result0 attack trials, and three of four boundaries checked here hold on evidence. The one that does not is the one this kit's shape invites: a register carries free text a third party wrote, and whether an instruction hidden in it could move what the model reports is unmeasured. The pure-code rulebook limits the blast radius — it reads typed values, not prose — but a changed date is still a changed status.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted contradiction is a REGISTER FLAG from the wrong register — a site calling an overdue condition 'on track' — which measures whether the pipeline runs the rulebook when the site's own paperwork points the other way. It does not measure whether a model obeys an instruction hidden in the same block. Those are different failures and only one of them is measured here.

HonestyWhat this does not prove

• Whether text placed inside a register's requirement line, condition-state parenthetical or waiver reference could move what the model reports. No attack run exists.
• Whether the kit behaves safely against a hostile provider — a response body crafted to break the JSON extraction in src/prompt.py::parse, or one that returns a plausible but fabricated extra condition block. parse() fails closed to an empty list, which the harness records as a failed document; a fabricated block would be counted as a spurious row by the scorer and would appear on the worklist in the app, and nothing beyond that was tested.
• Whether a real site's obligation register would carry anything sensitive this kit mishandles. The corpus has no personal data by construction; a real register carries named responsible officers, a site's actual breach history and correspondence with a regulator, and none of that path is exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

`escalate` fires when the register carries at least one obligation the rulebook makes OVERDUE whose own `register_flag` — the site's self-assessment of that row — reads `on track` or `closed`. Both values come out of the same reply; the rule is run afterwards in pure code, over whatever the model returned, and never over gold. A register with no readable date or no conditions returns None rather than False, and so does one whose overdue row carries a flag outside the vocabulary: on this rule an unknown is neither a pass NOR an alarm, because raising on an unreadable flag is a false alarm and clearing it is a silent one.

src/extract.py::compute(), called by extract() on every register, by the local app on every click, and — through escalate_from() — by evals/baseline.py on the free floor's own output, so the two are routed by identical code. The status rule it reads lives in a SEPARATE module, src/rulebook.py::decide(), which is also what the corpus generator wrote gold with and what src/prompt.py states to the model — one definition, three readers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a notification, a filing or a clearance. Nothing here submits anything to anybody, renews anything, or marks anything done, and status is a proposal with the rulebook's own reasoning attached rather than a determination anybody is bound by. A qualified person reads the permit.
• It is NOT monitoring. The kit reads one snapshot that somebody else assembled, as at the date printed on it. It does not poll, subscribe, schedule, alert or escalate to anyone, it holds no state between calls, and nothing in it runs unattended. If the register is a week stale, so is every status on this page.
• It is NOT a check on whether the extracted values are right. If the model misreads a date and the rulebook then computes a status consistent with the misreading, the register is escalated — or not escalated — on a wrong fact and this rule cannot tell. The register-flag diagnostic in evals/judge.py is the closest thing to that check, it is reported separately, and it is blind to exactly the same case.
• It is NOT any real escalation policy. Overdue-while-the-flag-is-quiet is this kit's own simplification, invented for this corpus. No permit, regulator's guidance or operator's procedure was consulted, and none is reproduced. A real compliance function weighs which condition lapsed, by how long, whether the breach is reportable in its own right, whether the administering body already knows, and who is authorised to disclose it.

WatchedWhat is watched, and why that one

The kit measures 14 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 7 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (failures 0, refusal_cells 0, thinking True) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

5 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

extraction · with the model in the path — 4 runs. Columns here are only ever compared with each other.

not a time series No two of these 4 runs measured the same system — they differ on documents, extraction_cells, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

extraction · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 7 chips that all say so.

DeviationsWhat deviated

0 breaches across 5 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/extract.py, on any change to data/rulebook.json (which moves the statuses the flag reads), on any change to data/fields.json's allowed values for register_flag, and on any corpus regeneration. Changing WHAT GETS ESCALATED is a policy change and must be re-scored, even though it never changes what is overdue.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the rulebook is a JSON file read with json, every due date is datetime.date plus timedelta, the provider is reached over urllib, and the UI is one HTML file with no build step. ⚠︎ AND IN PARTICULAR THERE IS NO SCHEDULER. A monitoring kit is the one place a reader might expect a cron library, a queue or a job runner, and a dependency there would be the first thing to suggest this kit watches something. It does not.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per register — segment, select, prompt, one call, parse, rulebook, worklist — with no branch, no loop and no agent. AND IT HAS NO SCHEDULER EITHER, which is the thing worth saying about a kit whose pattern is called monitor: there is no timer, no queue and no state between calls.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain: the JSON extraction from a fenced reply, the backoff policy, the section regex, the date parsing and the .env reader are all code somebody has to own. The date parsing in particular is deliberately strict — it refuses anything that is not YYYY-MM-DD — and a real register will hand you five other formats on day one.
• No framework means no framework's ecosystem — no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. THE LAST ONE IS NOT FREE HERE: this kit's reply is a nested object with a variable-length list, and one live read came back with one fewer entry than the register has. A schema library would not have caught that either — the reply was valid — but a structured-output mode that enforced a per-block enumeration might, and nothing here evaluates one.
• The provider abstraction covers exactly two wire formats. A third provider is one function and one dict entry, and until somebody writes it the kit runs on two shapes.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-permit-obligations on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 4 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b000-permit-obligations-flag, t000-permit-obligations-stub recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Every site name, site and permit identifier, administering office, condition identifier and waiver reference is invented, and no real operator, mine, site, permit, regulator or regulation is named. The obligation rulebook this kit ships (data/rulebook.json) is ILLUSTRATIVE and is not an authority. It was written for this kit and reproduces no permit, licence condition, regulator's guidance, industry code or statutory schedule. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-field exact match, light normalisation

Does the run's value for each field match gold, after trimming whitespace and punctuation? last_done, period_credited and stated_due are the three legitimately null fields — null where the register says the entry was logged with no date, where the line does not apply, or where no date is stated — so a null there is a hit, not a miss, when gold agrees. THE DENOMINATOR IS GOLD'S: a condition block the run never returned costs its eight cells rather than shrinking the denominator.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every run, and the same field-match logic the free floor is scored by — a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same values the register states; evals/check_labels.py asserts every non-nullable field is populated on every row, that condition ids are unique within a register, and — the strongest check available — that all 2,294 gold cells can be re-read off their own register by pure regex, before any run is allowed to spend.

How tight can the band be? There is no tolerance band — it is exact match after trimming whitespace and punctuation, never a continuous score to round.

Cadence: Re-score on any change to data/rulebook.json, src/prompt.py, data/fields.json or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe status, five-way and collapsed onto the worklist — the false-alarm rate

Two questions at once. Five-way: did the pure-code rulebook, run over what the model reported, land on the same status the rulebook produces from gold — overdue, due_in_window, not_yet_due, not_binding or not_determinable? And the one a monitoring desk actually acts on: OF EVERY ROW THIS RUN PUT IN FRONT OF A PERSON, WHAT SHARE DID NOT NEED TO BE THERE? That is the FALSE-ALARM RATE, and it is the headline rather than a footnote, because a queue that cries wolf is worse than no queue — a person clears every row on it by hand, every day. Its mirror, MISSED ACTIONS, is counted and named separately and never averaged into it.

MethodHow the test was run

evals/judge.py::score_statuses, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's status is re-derived inside the grader by the same lookup the kit publishes — src/rulebook.py::decide(), reading data/rulebook.json — so the truth this matrix grades against can never be a separately-typed label that drifted from the rule. The run's own answer is read off the run rather than recomputed, which is what makes this a grade of the shipped pipeline.

How tight can the band be? No threshold — the status is one of five values, and a row the run never returned is counted as unanswered rather than folded into a correct cell. On a monitoring queue that distinction is the whole difference between an obligation nobody raised and an obligation nobody needed to.

Cadence: Re-run whenever data/rulebook.json or src/rulebook.py changes, whenever the corpus is regenerated, and on any provider or model change.

The decisionWhen to reach for it

In one lineThe derived due date, on the rows where one is computed rather than read

Is the date right, not just the verdict? An obligation found but MIS-DATED is a different failure from one missed, and an accuracy figure cannot tell them apart: a row correctly called overdue with the wrong due date sends somebody to the wrong deadline. Scored only where the date is DERIVED — a cycle dated from the last recorded entry, or an annual report dated from the reporting period credited. Rows whose due date is stated on the register, and rows where the rule stops before any date applies, are excluded rather than counted as misses.

MethodHow the test was run

evals/judge.py::score_statuses, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/rulebook.py::decide()'s own due_date, computed from GOLD's values — the same arithmetic the run's own answer came out of, so a change to an interval moves both sides together.

How tight can the band be? No threshold — a date is right or it is not. Exact ISO string comparison.

Cadence: Re-run whenever data/rulebook.json's intervals or reporting deadline change.

The decisionWhen to reach for it

In one line“Cannot be determined from this record”, scored in both directions

How often does the kit reach for not_determinable, and how often is it right to? A monitor that never says it is not being cautious, it is guessing — and on a monitoring queue a confident wrong 'clear' is the failure that actually hurts. A monitor that says it everywhere is useless. Both directions are scored, and the class is pulled out of the five-way accuracy rather than left buried inside it.

MethodHow the test was run

evals/judge.py::score_statuses, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The same rulebook lookup over gold's values. The 26 rows are exactly the ones the register cannot answer: 16 cycle entries logged with no date and 10 triggers the register does not record either way.

How tight can the band be? No threshold — a binary collapse of the five-way status.

Cadence: Re-run on any corpus regeneration or rulebook change.

The decisionWhen to reach for it

In one lineThe escalation flag against the same rule run over gold

Does the pure-code escalation — something already overdue AND the site's own register flag quiet about it — land on the same registers it would land on if every field had been read perfectly? It is a BUSINESS CONDITION, so unlike a self-consistency check it genuinely needs labels, and saying so is half of what makes the number believable.

MethodHow the test was run

evals/judge.py::score_statuses, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute(), the same function the run uses, applied to GOLD's values. One rule, two callers, so a change to the rule moves both sides together and the grader cannot silently grade an old policy.

How tight can the band be? No threshold — a scan over the rows and a comparison.

Cadence: Re-run whenever compute() changes. Changing WHAT GETS ESCALATED is a policy change and must be re-scored, even though it never changes what is overdue.

The decisionWhen to reach for it