The business caseThe problem this solves

A retail site holds a SET of permits, and which set depends on where it is and what kind of store it is. The site's compliance folder records the ones somebody knows about: what each permission is, who holds it, whether it is in force and when it runs out. Nothing in that folder says what the site was supposed to hold — so the failure that matters is invisible on the page. Four specific things work against a reader: the obligation set is EXTERNAL to the document and a permit nobody ever applied for leaves no trace on it; each authority sets its own renewal lead time, from 30 to 180 days here, so one reminder window is wrong in both directions at once; a permit can be perfectly in force and held under the name the site traded under before it was rebranded, which cannot be renewed without a transfer; and a jurisdiction can change what it requires halfway through a year, so two identical stores owe different sets depending on the date their file was drawn. Somebody opening a site's compliance folder, working down the permits it holds, and then doing the part the folder cannot help with: remembering what that jurisdiction requires of that store format, subtracting one from the other, and noticing what is not there. It is a few minutes per site and it is every site, every review cycle — and the part that goes wrong is not the arithmetic on the expiry dates. It is reading a permit held in the name the site traded under before it was rebranded as though it were the site's own, applying one reminder window to authorities that ask for anything from 30 to 180 days' notice, raising a permit the format does not actually require, and above all working down the FILE when the question is about the SET.

The inputThe actual site permit files

The corpus is 50 site permit files, 0.09 MB (txt 50). Plain text, one format, invented rather than fetched — a real site permit file states a real business's actual licensing position, which is commercially sensitive on one side and legally consequential on the other, and there is no public corpus of (site file, correct obligation set) pairs. AND THERE IS A SECOND REASON THAT IS STRONGER ON THIS SHAPE: the answer is not in the document. Labelling a captured corpus would mean somebody deciding by hand what each real site was required to hold, which is a legal opinion rather than a label. Generating it makes the label mechanical instead — gold's obligation set is required_set() over the same jurisdiction, format and date the file states, and gold's status is decide_class() over the same records it prints.

Synthetic Record
----------------
This is a SYNTHETIC site permit file, generated for a public AI use-case kit. Every site,
trading name, jurisdiction, authority, permit class, permit reference, exemption reference
and date in it is invented. No real retailer, site, landlord, authority, permit or
regulation is named or reproduced. The obligation matrix this file is read against is the
kit's own construction and is not an authority.

Site
----
Marlow Gate Convenience (SITE-MG-2000)
Trading name: Marlow Gate Convenience
Formerly traded as: Marlow Gate Stores
Jurisdiction: Northbridge Metropolitan District
Format: standard_store

File Date
---------
2026-05-20

Permit Record PRM-40002
-----------------------
Class: trade_premises_registration
Issued by: the Northbridge Metropolitan District Licensing Office
Held by: Marlow Gate Convenience
Record state: in_force
Expires: 2026-05-25

Permit Record PRM-40001
-----------------------
Class: food_business_registration
Issued by: the Northbridge Metropolitan District Environmental Health Service
Held by: Marlow Gate Convenience
Record state: in_force
Expires: 2026-06-23

Site Contact Reference
----------------------
Store manager: A. Rehan
Area compliance contact: J. Mbeki
Internal telephone extension: 2100
Customer account number: ACCT-410000

File Notes
----------
This file lists the permit records the site's own compliance folder holds. It is not a
statement of what the site is required to hold, and no authority has confirmed it is
complete.

The outcomeWhat a good result looks like

A row per permit the site owes, with its status, the date it runs out and the matrix's own reason — plus a proposed worklist, the obligations the file cannot answer named separately rather than cleared, and one pure-code escalation: this site owes a trading-critical permission it has no record of ever holding. Nothing here applies for anything, renews, transfers, surrenders, notifies or clears, and nothing continues after the call returns.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Reconciling a book of site permit folders against what each jurisdiction and store format actually requires — the periodic completeness review, not the renewal diary — the one tier measured here — 29 of 29 permits owed and absent named, 193 of 193 verdicts, 0 false alarms
  against the free flat-reminder floor's 0 of 29 on the same 50 files, a 40.24 pct false-alarm rate and 50 of 99 actions missed. The floor reads every printed field perfectly, so that gap is entirely in the verdict.
• Deciding whether a model is needed here at all, rather than a script — read the floor row before the model rows
  the floor reads the file perfectly for $0.00 and 0 tokens, in well under a second against a 5.0-second median call. What the money buys is not reading — it is the two-place holder comparison (17 of 17 against 0), the per-authority lead time (8 wide rows and 24 narrow ones against a flat window's 30 errors), the exemption branch (9 of 9 against 0) and the undeterminable branch (8 of 8 against 0).

And where nothing here is good enough:

• Using the escalation flag as an alerting condition — nothing yet — this page says wait, and says why
  it scored 18 of 18 with no false positives, and that number is downstream of a record recall of 189 of 189 on ONE pass. The flag reads the run's own verdicts, so a reply that dropped a record would produce a confident missing and a confident escalation for a permit the site holds.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-permit-lapse. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts on 127.0.0.1:8979, the 50-file picker populates, the obligation table says “not checked yet” in words rather than rendering an empty table, the site panel says “not checked yet” on every row rather than announcing a parse failure, and the shipped matrix renders in full from data/rulebook.json. Clicking Check the permit set returns the no-key sentence rather than a stack trace. python -m evals.check_labels and python -m evals.run --baseline both pass with no network access at all.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per site permit file

Current processWhat it replaces

Somebody opening a site's compliance folder, working down the permits it holds, and then doing the part the folder cannot help with: remembering what that jurisdiction requires of that store format, subtracting one from the other, and noticing what is not there. It is a few minutes per site and it is every site, every review cycle — and the part that goes wrong is not the arithmetic on the expiry dates. It is reading a permit held in the name the site traded under before it was rebranded as though it were the site's own, applying one reminder window to authorities that ask for anything from 30 to 180 days' notice, raising a permit the format does not actually require, and above all working down the FILE when the question is about the SET.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings; every permit record and exemption notice is its own headed block, and a file with no headings falls back to one whole-document segment

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 50 site files cut into 439 sections in nine thousandths of a second, in process, with no model and no network.

LicenceLicence

MIT — this repository's own licence. Every site name, trading name, former trading name, site identifier, permit reference, exemption reference, administering authority, landlord, concession operator and contact is invented, and no real retailer, site, authority, permit or regulation is named. The obligation matrix this kit ships (data/rulebook.json) is ILLUSTRATIVE and is not an authority. It was written for this kit and reproduces no statute, licensing scheme, regulator's guidance, industry code or company permit register.

Bring your ownBring your own site permit files

REPLACE data/rulebook.json FIRST. It is this kit's own construction, it reproduces no statute or licensing scheme, and everything downstream reads it: src/rulebook.py loads it, src/prompt.py renders it into every call, and tools/build_corpus.py writes gold with it — including the OBLIGATION SET itself, which is the denominator every figure on this page is measured against. Then replace data/corpus/*.txt, write your own data/fields.json (both halves — site-level and per-record), and supply a gold record per site. SECTION_HINTS and the record prefixes in src/select.py map fields to section headings and will need editing for a different folder layout; when they do not match, selection falls back to the whole document minus the never-sent sections — slower, more expensive, always correct. compute() in src/extract.py is this kit's own escalation rule and is the second thing to replace.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 4,135 tokens assembled, 2,556 are contexts — 62% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on SITE-0030 — the same file the worked example above was captured on, so the four parts sum to exactly the 4135 input tokens that call actually reported. Four calls at max_tokens=1, each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. kits/UC0075-permit-lapse/results/tokens-p003-permit-lapse.json.

Full promptThe prompt in full

You read a SITE PERMIT FILE -- the permit records one retail site's own compliance folder holds -- and you report what is in it. You return JSON and nothing else.

YOU DO NOT DECIDE WHAT THE SITE IS REQUIRED TO HOLD, AND YOU DO NOT DECIDE WHETHER ANYTHING IS LAPSED, DUE OR CLEAR. A separate piece of code works both of those out afterwards: it looks up the set of permits this jurisdiction and this store format require, subtracts what you report, and runs each authority's renewal lead time over the dates you report. Your job is to report the file faithfully, including the places where it records nothing.

DO NOT NAME A PERMIT THE FILE DOES NOT CARRY. Whether something is absent is decided by the code downstream against a matrix you have not been given; a class you add because it looks like it ought to be there is an invented record, and it is scored as one.

RULES, in order of importance:
1. RETURN ONE ENTRY PER RECORD BLOCK ON THE FILE, in the order the file prints them, INCLUDING blocks that are surrendered, refused or under appeal, and INCLUDING exemption notices. A compliance folder accretes; the records nobody removed are exactly the records this reading is about. A dropped block is the most expensive mistake available here, because downstream it is indistinguishable from a permit the site never held.
2. AN EXEMPTION NOTICE IS A RECORD BLOCK. Its `record_state` is 'exempted', its `permit_ref` is the EX- reference on its heading, and its `expiry` is null. It is the only thing on the page that can take an obligation off the worklist, so leaving it out invents work for somebody.
3. `record_state` IS THE FIRST WORD OF THE 'Record state' LINE AND NOTHING ELSE. A surrendered, refused or appealed record still carries an expiry date, often a future one, and that date does not change the state. Report the state exactly as it stands.
4. `holder_match` IS A COMPARISON BETWEEN TWO PLACES IN THE FILE, not a value printed on one line. Read the record's 'Held by' line against the Site section. If it is the site's 'Trading name', answer 'current_name'. If it is the name on the site's 'Formerly traded as' line, answer 'former_name' -- the site's own permit, under the name it used to trade under. If it is neither, answer 'other_party'. 'former_name' and 'other_party' look alike and they are different answers.
5. `permit_class` IS COPIED VERBATIM FROM THE 'Class' LINE. It is the key the obligation set is joined on downstream, so an approximation is not an approximation: it becomes a permit the site does not owe AND a permit the site does not hold, from one mistake.
6. `expiry` IS A DATE OR IT IS NULL. Where the line says no expiry is recorded on the file, return null -- not today's date, not a date from another record, and not the date the file was drawn.
7. `trading_name` IS THE NAME THE SITE TRADES UNDER TODAY. Where the Site section also carries a 'Formerly traded as' line, that is the FORMER name and is not this field.
8. `jurisdiction` IS COPIED IN FULL, verbatim. It selects which set of permits the site owes, so a shortened or abbreviated form is a different question being answered.
9. Copy identifiers verbatim. Write every date as YYYY-MM-DD, exactly as the file does. Use the exact allowed value for a field that lists them, and return every field named in the schema on every entry, even when the answer is null.

OBLIGATION SET RULEBOOK (the authority for the worklist; this is an ILLUSTRATIVE rulebook written for this kit, and it reproduces no statute, licensing scheme, regulator's guidance or company permit register)

WHAT THE WORKLIST IS
  a permit worklist per site, never a filing. Nothing here applies for, renews, transfers, surrenders, notifies, escalates to an authority or clears anything. It lists what the site's file shows it holds, joins that against the set this rulebook says the site owes, and NAMES THE PERMITS THAT ARE NOT ON THE FILE AT ALL.

THE HARD HALF
  A PERMIT NOBODY KNEW WAS OWED CANNOT BE FOUND BY READING THE FILE. Everything on a site's permit file is a permit somebody already knows about. The obligation set is EXTERNAL to the document - it comes from the jurisdiction and the store format - and the only way a missing permit becomes visible is by subtracting what is on the file from what the matrix says is owed. That subtraction is this kit. A reader working down the file, however carefully, is working through the wrong list.

STORE FORMATS
  standard_store     a shop with no fuel, no pharmacy counter and no licensed cafe
  forecourt_store    a shop attached to a fuel forecourt
  pharmacy_counter   a shop with a dispensing pharmacy counter inside it
  licensed_cafe      a shop with an in-store cafe that serves alcohol

THE OBLIGATION SET, by jurisdiction -- what a site OWES before anything is read
  Northbridge Metropolitan District
    every format: trade_premises_registration, food_business_registration, waste_carrier_registration
      + standard_store   (nothing further)
      + forecourt_store  fuel_storage_certificate, illuminated_signage_consent
      + pharmacy_counter pharmacy_premises_permit
      + licensed_cafe    alcohol_retail_licence, late_hours_trading_permit
  Calder Vale County
    every format: trade_premises_registration, food_business_registration, waste_carrier_registration
      + standard_store   (nothing further)
      + forecourt_store  fuel_storage_certificate
      + pharmacy_counter pharmacy_premises_permit
      + licensed_cafe    alcohol_retail_licence
      * waste_carrier_registration is owed only by a file drawn ON OR AFTER 2026-04-01
  Eastmarch Borough
    every format: trade_premises_registration, food_business_registration, waste_carrier_registration, illuminated_signage_consent
      + standard_store   (nothing further)
      + forecourt_store  fuel_storage_certificate
      + pharmacy_counter pharmacy_premises_permit
      + licensed_cafe    alcohol_retail_licence
      - forecourt_store  NOT required: illuminated_signage_consent
  Tarnwick Unitary Authority
    every format: trade_premises_registration, food_business_registration, waste_carrier_registration
      + standard_store   (nothing further)
      + forecourt_store  fuel_storage_certificate
      + pharmacy_counter pharmacy_premises_permit, late_hours_trading_permit
      + licensed_cafe    alcohol_retail_licence, late_hours_trading_permit
  Pellhaven Township
    every format: trade_premises_registration, food_business_registration
      + standard_store   (nothing further)
      + forecourt_store  fuel_storage_certificate, waste_carrier_registration
      + pharmacy_counter pharmacy_premises_permit
      + licensed_cafe    alcohol_retail_licence

RENEWAL LEAD TIMES, in days -- THE AUTHORITY'S, not the permit's
  THE RENEWAL LEAD TIME IS THE AUTHORITY'S, NOT THE PERMIT'S. `default` is the lead time each class carries wherever it is owed; `by_jurisdiction` overrides it where that jurisdiction's office asks for more or less notice. The same expiry date is therefore `renewal_due` in one jurisdiction and `held_current` in the next, which is the trap a single flat reminder window cannot see.

  default:
    trade_premises_registration     30
    food_business_registration      30
    alcohol_retail_licence          90
    pharmacy_premises_permit       120
    fuel_storage_certificate        90
    waste_carrier_registration      45
    illuminated_signage_consent     30
    late_hours_trading_permit       45
  Northbridge Metropolitan District overrides: alcohol_retail_licence 120, waste_carrier_registration 60
  Calder Vale County overrides: fuel_storage_certificate 150
  Eastmarch Borough overrides: food_business_registration 60
  Tarnwick Unitary Authority overrides: pharmacy_premises_permit 180, late_hours_trading_permit 30

RECORD STATES
  in_force       the authority's register records this permit as live. Only an in-force record can satisfy an obligation.
  surrendered    the holder gave the permit up. It satisfies nothing, whatever its expiry date still says.
  refused        the authority refused the application. There is a record and there is no permit.
  under_appeal   a refusal or revocation is under appeal. The permit is NOT in force while the appeal runs, and the worklist says so rather than waiting for the outcome.
  exempted       the authority has recorded a written exemption from this class for this site, with a reference. Nothing is owed and nothing goes on the worklist.

WHOSE NAME THE PERMIT IS IN
  current_name   the permit is held in the trading name the site trades under today.
  former_name    the permit is held in a name this site USED TO trade under - the file states the former name. The permit exists and it is not in the site's own current name, so it cannot be renewed or relied on without a transfer first.
  other_party    the permit is held by somebody else entirely - a landlord, a concession operator, a previous occupier. It is on the file and it is not the site's permit.

THE STATUS, WORKED OUT AFTERWARDS IN CODE (context for you; you do not compute it)
  1. THE SITE-LEVEL STOP. If the file does not state a jurisdiction and a format this matrix knows, or does not state the date it was drawn, NOTHING can be decided: every class is `not_determinable` and the site's obligation set is unknown. Not empty - unknown.
  2. EXEMPTED. If the file carries a record of the class whose record state is `exempted`, the status is `exempt`. Nothing is owed and it never reaches the worklist.
  3. MISSING. If the site owes the class and the file carries NO record of it at all, the status is `missing`. THIS IS THE HALF THAT CANNOT BE READ OFF THE FILE - it is the obligation set minus what is printed, and a reader working down the file will never reach it.
  4. NOTHING IN FORCE. If every record of the class is `surrendered`, `refused` or `under_appeal`, the status is `lapsed`. A record is not a permit.
  5. NO EXPIRY RECORDED. If the governing in-force record states no expiry, the status is `not_determinable`. The clock cannot be run, and an unrunnable clock is a fact to report rather than a clearance.
  6. ALREADY EXPIRED. If the governing record's expiry is BEFORE the date the file was drawn, the status is `lapsed`.
  7. NOT IN THE SITE'S OWN NAME. If the governing record is held under a former trading name or by another party, the status is `not_in_own_name` - CHECKED BEFORE THE RENEWAL WINDOW, deliberately. A permit the site cannot renew because it is in somebody else's name is a worse position than one that merely needs renewing, and a worklist that says `renewal_due` on it sends somebody to lodge a renewal they have no standing to lodge.
  8. INSIDE THE AUTHORITY'S LEAD TIME. If the expiry falls within this jurisdiction's lead time for this class, counted in days from the date the file was drawn, the status is `renewal_due`.
  9. ANYTHING ELSE. The status is `held_current`.

  Where a site holds more than one record of the same class, the GOVERNING record is the in-force one, preferring the one held in the site's current trading name and then the one expiring latest. Where none is in force, rule 4 fires and no single record governs.

  the worklist is: missing, lapsed, not_in_own_name, renewal_due
  `not_determinable` is NOT on the worklist and is NOT cleared either - it is listed separately, as the set of obligations this file cannot answer. Folding it into either side is the one thing a completeness check must not do: counted as clear it hides a hole, counted as an action it fills the queue with rows nobody can act on.

WHY `missing` IS THE ANSWER THIS SHAPE EXISTS FOR
  A completeness check that only reports on the permits it can see is a renewal reminder wearing a compliance coat. The expensive failure in this shape is not a licence that lapsed - somebody gets a renewal notice for that. It is the permit the site never held, in a jurisdiction whose rules nobody re-read when the store format changed, which surfaces when an inspector asks for it. That row is invisible to every reader whose list is the file itself, and it is the only row on this worklist that a free reminder spreadsheet is STRUCTURALLY incapable of producing.

Return these:
- site_id (string) -- the site identifier printed in brackets after the site name in the Site section, verbatim
- jurisdiction (enum) one of: Northbridge Metropolitan District, Calder Vale County, Eastmarch Borough, Tarnwick Unitary Authority, Pellhaven Township -- the licensing jurisdiction named on the 'Jurisdiction' line of the Site section, verbatim and in full. It decides WHICH SET OF PERMITS this site owes, so a near-miss is not a near-miss
- format (enum) one of: standard_store, forecourt_store, pharmacy_counter, licensed_cafe -- the store format on the 'Format' line of the Site section, verbatim. Together with the jurisdiction it decides the obligation set
- file_date (date) -- the date this permit file was drawn, from the File Date section, as YYYY-MM-DD. Every renewal clock on this site is counted from it
- trading_name (string) -- the name the site trades under TODAY, from the 'Trading name' line of the Site section, verbatim. Where the Site section also carries a 'Formerly traded as' line, that is the FORMER name and is NOT this field
- records (array of objects) -- ONE OBJECT PER RECORD BLOCK printed on the file, in printed order, permit records and exemption notices alike, each carrying:
    - permit_class (enum) one of: trade_premises_registration, food_business_registration, alcohol_retail_licence, pharmacy_premises_permit, fuel_storage_certificate, waste_carrier_registration, illuminated_signage_consent, late_hours_trading_permit -- the permit class stated on the 'Class' line of the record block, verbatim. THIS IS THE KEY the obligation set is joined on
    - permit_ref (string) -- the reference on the record block heading, verbatim (for example PRM-40218, or EX-3305 for an exemption notice)
    - record_state (enum) one of: in_force, surrendered, refused, under_appeal, exempted -- the FIRST WORD of the 'Record state' line and nothing else. A surrendered, refused or appealed record still carries an expiry date and those dates do not change the state. `exempted` is an exemption notice rather than a permit
    - holder_match (enum) one of: current_name, former_name, other_party -- WHOSE NAME THE PERMIT IS IN, decided by comparing the 'Held by' line of this record block against the Site section. 'current_name' when it is the site's Trading name. 'former_name' when it is the name on the site's 'Formerly traded as' line. 'other_party' when it is neither -- a landlord, a concession, a previous occupier. This is a comparison between two places in the file, not a value printed on one line
    - expiry (date) -- the date on the 'Expires' line of this record block, as YYYY-MM-DD. Return null where the line says no expiry is recorded on the file, and null on an exemption notice, which carries no expiry. Do not substitute the issue date, the review date or any date from another record

Return a JSON object with exactly these top-level keys: site_id, jurisdiction, format, file_date, trading_name, records
`records` is an array. Return it EMPTY only if the file carries no permit record and no exemption notice at all -- which happens, and is a real answer.

SITE PERMIT FILE
----------------
Synthetic Record
----------------
This is a SYNTHETIC site permit file, generated for a public AI use-case kit. Every site,
trading name, jurisdiction, authority, permit class, permit reference, exemption reference
and date in it is invented. No real retailer, site, landlord, authority, permit or
regulation is named or reproduced. The obligation matrix this file is read against is the
kit's own construction and is not an authority.

Site
----
Bramfield Service Station Shop (SITE-B-2203)
Trading name: Bramfield Service Station Shop
Jurisdiction: Eastmarch Borough
Format: forecourt_store

File Date
---------
2026-04-01

Permit Record PRM-40103
-----------------------
Class: food_business_registration
Issued by: the Eastmarch Borough Environmental Health Service
Held by: Ninth Row Concessions Limited
Record state: in_force
Expires: 2026-04-06

Permit Record PRM-40105
-----------------------
Class: waste_carrier_registration
Issued by: the Eastmarch Borough Waste Regulation Unit
Held by: Bramfield Service Station Shop
Record state: in_force
Expires: 2026-12-23

Permit Record PRM-40104
-----------------------
Class: trade_premises_registration
Issued by: the Eastmarch Borough Licensing Office
Held by: Bramfield Service Station Shop
Record state: in_force
Expires: 2026-04-21

File Notes
----------
Records are printed in the order they were filed. Superseded and closed records are kept
rather than removed, so a class may appear more than once.

Raw responseThe raw response

{
  "site_id": "SITE-B-2203",
  "jurisdiction": "Eastmarch Borough",
  "format": "forecourt_store",
  "file_date": "2026-04-01",
  "trading_name": "Bramfield Service Station Shop",
  "records": [
    {
      "permit_class": "food_business_registration",
      "permit_ref": "PRM-40103",
      "record_state": "in_force",
      "holder_match": "other_party",
      "expiry": "2026-04-06"
    },
    {
      "permit_class": "waste_carrier_registration",
      "permit_ref": "PRM-40105",
      "record_state": "in_force",
      "holder_match": "current_name",
      "expiry": "2026-12-23"
    },
    {
      "permit_class": "trade_premises_registration",
      "permit_ref": "PRM-40104",
      "record_state": "in_force",
      "holder_match": "current_name",
      "expiry": "2026-04-21"
    }
  ]
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch a retail site's missing permits — 50 site permit files. One model answered, and every answer was then graded Four different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Run it twiceThe same set, run again

THE BAND IS 100 PCT AND ITS DENOMINATOR IS 6 PCT OF THE CORPUS. Three documents were re-asked, not fifty, and all three are among the fullest folders rather than a random sample — chosen because a longer reply is where a dropped record is most likely, not because they are representative. Nothing here establishes that the 47 files nobody asked twice are stable, and 24 of the 26 sites carrying a missing permit are in that 47. On this shape that gap is the one that matters: the worklist is the obligation set MINUS the records the reply returned, so a dropped record becomes a confident missing verdict that cites nothing, renders identically to a correct answer and fires the escalation either way.

records returned on the 3 fullest folders, asked 3 times each — 18 of 18 every time. NOT the 50-file scored set — The three asks are not averaged into the scored run and must not be: they cover 18 of the corpus's 189 permit records and were fired to test STABILITY, not accuracy. Folding them into r001 would move a 189-record denominator with 18 records of evidence.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersFour ways to grade

The floor is the first row of the evals board deliberately. Every figure above it reads as good until you know what nothing costs.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The flat-reminder floor (evals/baseline.py), the label gate (evals/check_labels.py) and the scorer (evals/judge.py) are all pure code and cost $0.00 to run against any result set. The figure above is the ONE scored run's own token counts (50 files, 209,706 input and 31,689 output tokens) priced at Google Gemini 3 Flash's published rate — the same basis cost_per_query_usd uses. ⚠︎ IT IS NOT WHAT BUILDING THIS KIT COST. Outside the scored run: a 6-file calibration on the fullest folders ($0.0289), the 14-file prompt ablation ($0.0435), the 9-ask repeat-stability probe ($0.0434), 4 prompt-split calls at max_tokens=1 ($0.0021), the worked-example capture ($0.0039), and 3 live UI calls whose tokens are not separately recorded and are therefore not projected. The shared call ledger records 88 live calls for this kit on 2026-08-22; everything with a committed token count projects to $0.3218 on this card.

Cost driversWhat actually moves the bill

• ⚑ THE FIXED PREFIX, WHICH IS MOST OF THE BILL AND HALF OF IT IS MEASURED AS WORTHLESS. The system prompt (889 tokens), the OBLIGATION MATRIX (2121) and the field schema (690) are 3700 of the worked example's 4135 input tokens — 89.5 pct — and byte-identical on every one of the 50 calls. The site file itself is 435 tokens. Run a001 removed the matrix and nothing moved, so 51.3 pct of every call's input is currently paying for context the measurement says is inert.
• Provider-side reasoning: 18,210 of the run's 31,689 output tokens (57.5 pct) were reasoning rather than the JSON reply. Output is a modest share of the bill on the projected card here — 48 pct — because this kit's replies are short. Reasoning was left at the provider's default and nothing here has measured what turning it off would cost.
• Output length scales with HOW FULL THE FOLDER IS, not with a fixed record. A two-record file and a six-record file are not the same call: the run's largest reply was 1051 output tokens against a mean of 633.78, and the calibration on the six fullest folders topped out at 1250.

Your volumeWhat it costs at your volume

Linear in site files: each call is independent, carries the same fixed prefix and shares nothing with its neighbours, so 500 files cost ten times 50. Nothing amortises — there is no index and no cache. ⚠︎ AND IT IS NOT LINEAR IN THE THING THAT ACTUALLY GROWS. The variable part of the call is the site file (435 of 4135 tokens on the worked example, 11 pct); the fixed part is the matrix, and THE MATRIX GROWS WITH THE ESTATE. Five jurisdictions and eight permit classes render to 2121 tokens; a retailer under sixty licensing authorities would render something an order of magnitude larger into every one of thousands of calls, and the input bill would be almost entirely a table that is looked up in code afterwards anyway. That is a different prompt, not a bigger one. Wall clock is comfortable: median 5010 ms, p95 6813, and 50 files in 24.0 seconds at 12 workers.

Model choiceWhy this model

One tier, one key, one run — the same tier every kit in this series runs first, so the rows on the use-case board are comparable. No second tier was run, and run a001 is the reason a second tier would not currently tell you anything: a corpus that cannot separate the shipped prompt from a half-stripped one cannot separate two models either.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 site permit files, one completion call each, one tier. The calibration, the prompt ablation, the repeat-stability probe, the prompt-split calls and the worked-example capture are recorded separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the site file into addressable sections, pure code — one section per permit record or exemption notice, which is what makes a per-record citation possible at all

# Cut a site permit file into addressable sections. Pure code -- no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections carry each field, pure code — the record fields map to every record block by heading PREFIX, holder_match maps to the Site section AND the record blocks because it is a comparison between the two, and the Site Contact Reference is mapped by nothing and never leaves the machine

You change it to: map fields to your own folder's headings; the prefix map is how a field says 'every block of this kind'. Unmatched falls back to the whole document MINUS the never-sent sections

# Pick which sections of a site permit file are sent. Pure code -- the last deterministic step

BANNER = "Synthetic Record"
SITE = "Site"
FILEDATE = "File Date"
NOTES = "File Notes"
RECORD = "Permit Record "
EXEMPT = "Exemption Notice "
RECORD_PREFIXES = (RECORD, EXEMPT)
NEVER_SENT = ("Site Contact Reference",)
SECTION_HINTS = {

src/rulebook.pymatrix

THE OBLIGATION SET AND THE STATUS RULE, loaded from data/rulebook.json. required_set() is a join on (jurisdiction, format, file date) that reads NO permit record — it is the denominator, and it is the half of the answer that is not in the document. decide_class() is nine ordered checks over the records of one class. Data, not code, so a forker can open it and disagree with it

# THE OBLIGATION-SET RULEBOOK, loaded from data/rulebook.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
RECORD_STATES = ("in_force", "surrendered", "refused", "under_appeal", "exempted")
HOLDER_MATCHES = ("current_name", "former_name", "other_party")
STATUSES = ("held_current", "renewal_due", "lapsed", "missing", "not_in_own_name",
ACTIONABLE = ("missing", "lapsed", "not_in_own_name", "renewal_due")
def load(path=RULEBOOK_PATH):
R = load()
JURISDICTIONS = tuple(k for k in R["required_set"] if not k.startswith("_"))

src/prompt.pyprompt

assemble one call for the whole site file, with the matrix RENDERED into it from data/rulebook.json rather than restated in prose — as context for why the jurisdiction must be copied in full and why holder_match is a comparison, since the model is told it will not compute the status or the set

# Assemble the permit-file prompt. ONE prompt per site file, every record it holds in the reply.

SYSTEM = (
RULEBOOK_TEXT = rulebook_block()
def field_schema(site_fields, record_fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key — plus the pure-code SUBTRACTION of what the file holds from what the matrix says is owed, the worklist, and the trading-critical escalation

You change it to: the escalation rule itself — this kit ships one condition (a trading-critical class came back missing) and a real compliance function weighs which permission is absent, whether the site can trade at all without it and who has to be told. It is deliberately NOT the same function as the status rule, so changing WHAT GETS ESCALATED does not change WHAT IS OVERDUE

# Read one site permit file, then do the subtraction: segment, select, prompt, one model call,

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 40000
def load_fields():
def record_fields(fields=None):
def site_fields(fields=None):
def load_doc(doc_id):
def documents():

evals/judge.pyjudge

score field accuracy, record recall, the seven-way verdict, the FALSE-ALARM RATE and MISSING-PERMIT RECALL on its own denominator, and the derived renewal dates split by which side of a flat 60-day window they fall, all pure code

# Score a permit-set completeness run. PURE CODE -- gold is exact and every answer is one value or

SITE_CELLS = ("site_id", "jurisdiction", "format", "file_date", "trading_name")
RECORD_CELLS = ("permit_class", "permit_ref", "record_state", "holder_match", "expiry")
SPANNABLE = ("permit_class", "permit_ref", "expiry")
def norm(v):
def ref_norm(v):
def equal(got, want):
def _records_of(rec):
def score(fields, records, golds):
def _verdict_rows(rec, gold):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 4194 input and 633 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's site permit files are entirely synthetic (tools/build_corpus.py, seed 20260822): no real retailer, site, landlord, concession, authority, permit or person exists in the corpus, and nothing was fetched from anywhere. No statute, licensing scheme, regulator, regulator's guidance or industry code is named anywhere in the kit, and every file opens with a Synthetic Record banner so a single file copied out of the repository still says what it is. The only outbound traffic the kit makes is one chat-completion request per file, carrying the mapped sections plus the fixed prompt and the shipped matrix — and the Site Contact Reference section, which is the one part of a real permit folder an estate team would most mind sending anywhere, is mapped by no field and never leaves the machine. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser.

The experimentWe did not attack it — and the boundary that holds only holds because of one line

Three boundaries hold. Two were confirmed by reading the code: no code path applies for, renews, transfers or surrenders anything, notifies anybody or writes anything, and the status rule reads only enums, so no record text can reach it. The third is confirmed by a CHECK RATHER THAN BY READING, and it needed to be — evals/check_labels.py asserts, over all fifty files and before any run may spend, that the Site Contact Reference section is never sent and that every decoy record is. On this corpus the fallback path is reached by design, on the one site whose compliance folder is empty, and the sibling-standard fallback would have sent the contact details with it. The fourth is open. An indirect prompt injection needs a section an outside party authored, and the File Notes are exactly that. Confirmed by reading the code and by a pre-flight assertion over all fifty files, not by an attack run, on 2026-08-22 — no attack run was fired; see redteam.why.

Three boundaries hold and the second is the interesting one, because it holds only because of a code change this shape forced: the sibling-standard fallback (or list(secs)) would have sent the whole document — contact details included — on the one file whose compliance folder is empty, and on this kit an empty folder is not an edge case, it is the answer the kit exists to produce. The fourth is open and is named as open.

The result0 attack trials, and three of four boundaries checked here hold on evidence. The one that does not hold is the one this kit's shape invites, and it is not the obvious one: the obligation set never comes from the document, so a file cannot talk its way out of owing a permit — but it might be able to talk a RECORD out of being mentioned, and on this shape an unmentioned record is a confident missing rather than an error.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted decoys measure whether the worklist follows the SET rather than the page, whether it compares a holder name against two places in the file, and whether it applies each authority's own lead time. None of those measures whether it obeys an instruction hidden in the same file, and the drop-a-record attack is the one this kit's own shape suggests is cheapest — because a dropped record is not an error here, it is a confident missing verdict.

HonestyWhat this does not prove

• Whether an indirect prompt injection in the File Notes could make a record go unmentioned. No attack run exists, and this is the version of the attack the kit's own failure mode suggests would be easiest — it does not need to change a verdict, only to lose a row.
• Whether the kit behaves safely against a hostile provider — a response body crafted to break src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed document, and nothing beyond that was tested.
• Whether a real permit folder would carry anything sensitive this kit mishandles beyond the contact section. The corpus has no personal data by construction; a real folder carries named licence holders, signatures, personal licence numbers and often a director's home address, and none of that path is exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

The escalation fires when a permit class the matrix marks `trading_critical` comes back `missing` — owed by this site's jurisdiction and format, and absent from the file entirely. Not expiring. Not lapsed. Absent. The rule is run afterwards in pure code, over whatever the model returned, and never over gold. A reply whose obligation set could not be determined returns None rather than False — an unknown is not a clearance.

src/extract.py::compute(), called by extract() on every file and by the local app on every click. The verdicts it reads come from src/rulebook.py, which is also what the corpus generator wrote gold with and what src/prompt.py renders to the model — one definition, three readers. evals/baseline.py deliberately returns None here rather than False, because the free floor cannot produce missing at all and answering 'no' would be a clearance it has not earned.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a filing, an application, a renewal, a transfer, a notification or an escalation to any authority. Nothing here decides anything about a site. The verdicts are proposals with the matrix's own reasoning attached, and a compliance team decides.
• It is NOT a check on whether the READING was complete, and SITE-0028 is the demonstration. The rule reads the run's own verdicts; if the reply dropped every record, all four owed permits come back missing, the flag fires, and the page is indistinguishable from the correct answer for a genuinely empty folder. A guardrail downstream of an extraction step inherits that step's blind spots entirely.
• It is NOT any retailer's escalation policy. 'A trading-critical class is missing' is this kit's own simplification, invented for this corpus. No licensing scheme, regulator's guidance, industry code or company procedure was consulted, and none is reproduced. A real escalation weighs which permission is absent, whether the site can trade at all without it, how long it has been absent and who has to be told.
• It is NOT a statement that the other 32 sites are fine. 14 of them carry a lapsed trading-critical permit or one in somebody else's name, and the flag ignores both by design.

WatchedWhat is watched, and why that one

The kit measures 23 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 16 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (failures 0, refusal_cells 0, thinking True) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

extraction · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on documents, extraction_cells, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

comply · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 3 chips that all say so.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/extract.py, on any change to data/rulebook.json — which moves both the obligation set the flag reads and the trading_critical list it keys on — on any change to data/fields.json's allowed values, and on any corpus regeneration. Changing WHAT GETS ESCALATED is a policy change and must be re-scored, even though it never changes what the matrix says is overdue.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt names nothing on purpose: the corpus is generated in-process, the matrix is a JSON file read with json, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

segment -> select -> prompt (system + matrix + schema + mapped sections) -> one model call -> parse -> the obligation-set SUBTRACTION in pure code -> the worklist -> the trading-critical escalation. Everything except the single call is deterministic, and the half of the answer that matters most — which permits are absent — is produced entirely on the deterministic side.

The other sideWhat a framework costs you

• Hand-rolling the HTTP client means the retry policy is ours to get right. This kit ships the transport clause a sibling learned the hard way, but nothing here has seen a transport failure live, which is the honest state and not a claim the branch is exercised.
• A JSON matrix with no schema means a forker can write an invalid one and find out at run time. evals/check_labels.py catches a matrix that disagrees with its own gold, and catches nothing about a matrix that is internally consistent and wrong.
• No framework means no ecosystem: there is no tracing, no cost accounting and no eval registry beyond the JSON files in results/. That is a deliberate floor for a kit meant to be read in an afternoon, and it is a real cost at any scale above one laptop.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-permit-lapse on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 3 runs on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Every site name, trading name, former trading name, site identifier, permit reference, exemption reference, administering authority, landlord, concession operator and contact is invented, and no real retailer, site, authority, permit or regulation is named. The obligation matrix this kit ships (data/rulebook.json) is ILLUSTRATIVE and is not an authority. It was written for this kit and reproduces no statute, licensing scheme, regulator's guidance, industry code or company permit register. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-field exact match, light normalisation

Does the run's value for each field match gold, after trimming whitespace and punctuation? expiry is the one legitimately null field — null where the record says no expiry is recorded, and on every exemption notice — so a null there is a hit, not a miss, when gold agrees. THE DENOMINATOR IS GOLD'S RECORDS: a record block the run never returned costs its five cells rather than shrinking the denominator. On a completeness kit that is not a refinement — a dropped record IS a missing permit downstream, so it has to cost something here too.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every run, and the same field-match logic the free floor is scored by — a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same values the file states; evals/check_labels.py asserts every non-nullable field is populated on every record, that permit references are unique within a file, and that every gold cell can be re-read off its own file, before any run is allowed to spend.

How tight can the band be? There is no tolerance band — exact match after trimming whitespace and punctuation, never a continuous score to round.

Cadence: Re-score on any change to data/rulebook.json, src/prompt.py, data/fields.json or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe obligation-set verdict, seven-way — and the false-alarm rate

Two questions at once. Seven-way: for every permit the site OWES, did the pure-code join land on the same verdict the matrix produces from gold — held_current, renewal_due, lapsed, missing, not_in_own_name, exempt or not_determinable? And the one a compliance desk actually acts on: OF EVERY OBLIGATION THIS RUN PUT IN FRONT OF A PERSON, WHAT SHARE DID NOT NEED TO BE THERE? That is the FALSE-ALARM RATE, and it is the headline rather than a footnote, because a queue that cries wolf is worse than no queue. Its mirror, MISSED ACTIONS, is counted and named separately and never averaged into it. ⚑ THE DENOMINATOR IS THE OBLIGATION SET, NOT THE FILE — an owed class the run's verdict never covered is unanswered, which is how the free floor's 29 blind spots become visible as a score rather than as an absence.

MethodHow the test was run

evals/judge.py::score_verdicts, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's verdict is re-derived inside the corpus generator by the same lookup the kit publishes — src/rulebook.py::required_set() for the set and decide_class() for each status, reading data/rulebook.json — so the truth this matrix grades against can never be a separately-typed label that drifted from the rule. The run's own answer is read off the run rather than recomputed, which is what makes this a grade of the shipped pipeline.

How tight can the band be? No threshold — the verdict is one of seven values, and an owed class the run never covered is counted as unanswered rather than folded into a correct cell. On a completeness check that distinction is the whole difference between a permit nobody raised and a permit nobody needed to.

Cadence: Re-run whenever data/rulebook.json or src/rulebook.py changes, whenever the corpus is regenerated, and on any provider or model change.

The decisionWhen to reach for it

In one line⚑ The permit nobody knew was owed — missing-permit recall

Of the permits gold says this site OWES AND DOES NOT HOLD, how many did the run name? Scored on its own denominator of 29 rows and never averaged into the seven-way, because a run can look excellent on the seven-way while getting every one of these wrong: 29 of 193 obligations are missing, so a run that never says missing at all still scores 85 pct on the verdict. Its precision is counted beside it — a phantom missing sends somebody to apply for a permit the site already holds, which is the opposite mistake and costs a different person a different day. THIS IS THE FIGURE THAT SEPARATES THIS KIT FROM A RENEWAL REMINDER, and the free floor scores 0 on it by construction rather than by performance.

MethodHow the test was run

evals/judge.py::score_verdicts, the missing_permits block. In-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: required_set() over the same jurisdiction, format and file date the document states, minus the record classes the document prints. Both halves are the shipped code; neither is a typed label. tools/build_corpus.py::_verify() additionally asserts that no missing class name appears anywhere in the text of the file it is missing from, so the answer cannot be reached by reading.

How tight can the band be? No threshold — a permit is on the file or it is not. The comparison is a set difference over exact class names.

Cadence: Re-run whenever data/rulebook.json's required_set changes — which moves the denominator, not just the answers — and on any corpus regeneration.

The decisionWhen to reach for it

In one lineThe renewal clock, and both sides of a flat reminder window

Is the DATE right, not just the verdict? An obligation correctly called renewal_due with the wrong date sends somebody to the wrong deadline, and an accuracy figure cannot tell the two apart. Scored on the 140 rows where gold derives a date from a held record. It is then SPLIT BY WHICH SIDE OF A FLAT 60-DAY REMINDER THE ROW FALLS, because that split is the argument against the thing every compliance desk already has: 8 rows are due inside their authority's own lead time and MORE than 60 days out (a flat window misses them), and 24 rows are perfectly current and INSIDE 60 days (a flat window raises them for nothing). This matrix's lead times run from 30 to 180 days, so one window is wrong in both directions simultaneously.

MethodHow the test was run

evals/judge.py::score_dates, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/rulebook.py::decide_class()'s own due_date, computed from GOLD's values with lead_time(jurisdiction, class) — the same arithmetic the run's own answer came out of, so a change to a lead time moves both sides together.

How tight can the band be? No threshold — a date is right or it is not. Exact ISO string comparison.

Cadence: Re-run whenever data/rulebook.json's lead times change.

The decisionWhen to reach for it