The business caseThe problem this solves

A work authorisation package lands on a permit desk before a shift -- a permit form with labels down the left margin, a system export, a set of CSV blocks, or a shift handover written in sentences -- and somebody has to satisfy themselves that every control the declared hazards require is actually in place before a crew goes to work. Almost none of that is a judgement. It is a join (which controls does this hazard require), a subtraction (is this gas test still fresh at the stated start), a comparison (is this certificate still valid) and a set-membership test (does this signature cover this hazard) -- in front of prose that says "spaded and the lock is on" and "still passing and sits under the flushing certificate". The supervisor's manual walk of the package against the rulebook -- deriving the checklist from the declared hazards, subtracting each gas test's age against the stated start, comparing each certificate's expiry, reading each isolation's state and testing each signature's scope -- before a person confirms the report.

The inputThe actual work authorisation packages

The corpus is 60 work authorisation packages, 0.06 MB (txt 60). A defect mix a permit office would recognise and no public dataset can provide: an isolation listed and left open under a flushing certificate; a package signed by two people whose authorisations cover hot work and work at height on a job that also declares energised electrical work; a valve found open on the walk-down and not re-set; a flammable reading of 8.6 pctLEL on a confined-space entry presented as ready to start against a 5.0 limit; an electrical isolation on a package that never declared electrical work. Five sixths are planted because a clean package teaches nothing about the arithmetic, and a sixth are clean because a corpus that is all traps measures a different job. ⚑ AND THE CORPUS WAS DELIBERATELY MADE HARDER MID-BUILD -- this is the clearest recorded case of it in the estate. The FIRST version of the prose renderer wrote the handover note out of the rulebook's OWN hazard labels and its three state words, and the free rules floor then scored 100.0 pct of control verdicts on all fifteen prose packages and 99.4 pct OVERALL. A corpus whose "sentences" are field labels with punctuation around them cannot tell a reader from a regex, and a kit built on one has not measured a reader. The notes were rewritten into ordinary language -- "a crane lift", "we are going inside the vessel", "spaded and the lock is on", "still passing and sits under the flushing certificate", none of it in data/rulebook.json -- and the floor fell to 23.4 pct on the prose format and 80.2 pct overall. BOTH NUMBERS ARE PART OF THIS KIT'S EVIDENCE: 99.4 pct is what the floor scores against a corpus that separates nothing, and 80.2 pct is what it scores against this one. The separation this kit reports was manufactured on purpose, and a reader is owed that before they read the 23.4-to-98.6 headline. ⚠︎ data/SOURCES.md records the post-rewrite floor as 78.2 pct in its prose while the shipped result file and the README both measure 80.2 pct (449 of 560). The 80.2 pct is the measured figure and is what this spec publishes; the 78.2 in SOURCES.md is stale and is recorded here rather than silently reconciled.

PRE-JOB BRIEFING NOTE  --  PK-0001

Job on Alderoak Gas Plant, crude storage tank T-104: internal shell inspection after the wash. On the hazard side we have a break-in on a system that holds pressure and work in the amine. The authorisation runs from 08:00 on the 17th of September 2026 and it closes at 18:00 on the 17th of September 2026; all times in this note are UTC.

On isolations: the pressure point at PR-474 is isolated and the lock is hung.

No gas test is written on the sheet.

Paperwork on file: the depressurisation record is DP-3516; the substance data sheet is SD-2224.

Signatures: H. Dumas (responsible electrical person) signed at 07:50 on the 17th of September 2026 and is authorised for a break-in on a system that holds pressure and there is a hazardous substance in the line.

Crew are due on the job at 10:00 on the 17th of September 2026.

The outcomeWhat a good result looks like

A pre-start report a person confirms instead of a permit a person re-reads: one row per control the declared hazards require, in the rulebook's fixed order, each SATISFIED, MISSING, EXPIRED or INCONSISTENT, each failure naming the rule id it breaks -- R-ISO-02, not "isolation problem" -- and one sentence saying what decided it. may_start is false when any control is not SATISFIED.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Packages that come out of a permit system as labelled fields -- a permit form, a CSV export, a checklist export — the free rules floor alone
  415 of 415 control rows and 45 of 45 packages, for $0.00, in 0.1 s. The model's own report on those same formats is 405 of 415 -- WORSE -- and reaches 415 only after pure code re-derives its verdicts. The best the paid arm managed on three quarters of this corpus was a tie.
• Shift handovers, pre-job briefings, anything where the hazards and the isolation states are sentences rather than fields — the model, rechecked
  23.4 pct to 98.6 pct on 145 control rows and 0 to 11 of 15 packages. The floor recovers 7 of 35 declared hazards across the fifteen notes and never builds the rows those hazards require: 93 rows go unreported, 16.6 pct of the corpus and 64.1 pct of the rows the prose packages carry -- including seven failing controls nobody is ever told about. That row, seven packages wide, is this kit's whole case for spending anything.
• A mixed queue -- some packages from the system, some typed up by a supervisor — the floor first, the model only on what the floor cannot declare
  The floor's failure mode is legible and cheap to detect: its DEC-CONS row fails on 13 of the 15 prose packages (11 of them wrongly) and on none of the 45 labelled ones, and on the 9 where it recovered no hazards at all it says so in as many words -- "the package declares no hazards at all, so there is no checklist to compare it against". That is the signature of a reader that failed rather than a package that is wrong. Route those to the model and pay for 15 packages instead of 60.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-31. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-31 — r001-worksafe-precheck. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — ⚠︎ NOT RE-MEASURED FOR THIS SPEC -- stated from what ships in the repo, not from a fresh keyless clone timed on this machine. What is on disk and verifiable without a key: data/corpus/ (60 packages), data/packages.json, data/rulebook.json, data/gold.jsonl, data/corpus-stats.json, the free rules-floor run (results/eval-b000-worksafe-precheck-rules.json, 449 of 560 control rows, 0.1 s wall, usd null), the stub run (results/eval-t000-worksafe-precheck-stub.json), the scored paid run and its per-call cache (results/eval-r001-worksafe-precheck.json and cache-r001-worksafe-precheck.jsonl). requirements.txt names NOTHING -- the kit is Python standard library end to end -- and .env is gitignored from the first commit, so the repo has never held a credential. python3 -m tools.build_corpus, python3 -m evals.check_labels, python3 -m evals.run --floor rules and python3 -m src.app are the four free commands the README publishes. ⚠︎ .env.example still says "NO PAID ARM IS COMMITTED FOR THIS KIT"; that line predates r001 and is stale, and it is recorded here rather than repaired from this side of the fence.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one pre-start report -- one package, one model call, end to end including provider-side reasoning tokens, on a shared connection. The recheck adds nothing measurable: it is pure Python over the reply. ⚠︎ NOT AN SLA: this credential is shared with sibling kits and these figures are a wall-clock observation of a single run. 60 packages ran in 967.5 wall seconds (16.1 minutes) at 5 concurrent workers, against 4,555.2 seconds of summed call time. LATENCY, NOT MONEY, IS THE CONSTRAINT HERE: 37 cents for sixty packages is nothing and a p95 of 215.3 s per package is what decides whether this runs before a shift or after it. The free rules floor answers all 60 in 0.1 s of wall clock, single-threaded, with no network at all (p50 0 ms, p95 1 ms).

Current processWhat it replaces

The supervisor's manual walk of the package against the rulebook -- deriving the checklist from the declared hazards, subtracting each gas test's age against the stated start, comparing each certificate's expiry, reading each isolation's state and testing each signature's scope -- before a person confirms the report.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one control rows on 60 pre-start report is

No train/test split, because nothing is trained or tuned, and no segmentation, because one package goes whole into one call (the sizes here are the packages' own bytes). THE UNIT EVERY RATE STANDS ON IS THE CONTROL ROW, not the package: a report that gets eight of nine rows right is one row wrong, and which row decides whether a crew goes to work. 560 rows over 60 packages. 10 packages are clean; the other 50 each plant a named thing across 13 cases -- 6 control_missing, 3 two_missing, 3 signoff_missing, 5 reading_stale, 4 cert_expired, 3 window_closed, 5 signoff_unscoped, 5 isolation_open, 3 isolation_conflict, 5 hazard_undeclared, 4 reading_out_of_limit and 4 mixed_faults -- and the generator refuses to write a package in which the mutation did not produce the verdict its case name promises, on the control it aimed at.

SetupWhat the setup figure measured

There is no index and no retrieval step. The rulebook and one package go whole into one prompt at 4,039.3 input tokens on average. The 0.1 s is the free floor's own measured wall clock over all 60 packages, single-threaded, no key and no network (results/eval-b000-worksafe-precheck-rules.json: wall_seconds 0.1, p50 0 ms, p95 1 ms).

LicenceLicence

MIT, with the corpus, the rulebook and the answer key all generated in-process -- there is no third-party data in this kit to licence.

Bring your ownBring your own work authorisation packages

Drop your own packages as .txt files into data/corpus/ and add one desk-record row per package to data/packages.json -- the package id, the asset, the job and THE MOMENT WORK IS DUE TO BEGIN, which is the one thing that must not be read out of the document (a package whose own window has closed still says it is valid). Your scheme goes in data/rulebook.json: hazards, controls, rule ids, freshness windows, atmospheric limits, isolation state words and the verdict precedence. The engine, the floor, the recheck and the prompt all read that one file, so a scheme change is a data change and the prompt cannot drift from the engine. Without a gold.jsonl of your own the UI works and the eval has nothing to score against.

⚠︎ And what stops being true when you do: ⚠︎ A REAL WORK AUTHORISATION PACKAGE REACHES YOUR CONFIGURED PROVIDER VERBATIM. It names the people who signed it, their authorisations, the equipment they isolated and the hazards somebody accepted on behalf of a crew -- one company's safety record and several people's employment files at once, and evidence if the job goes wrong. THIS KIT HAS NO REDACTION STEP and the seam that protects the scheme (data/rulebook.json) protects nothing in the package's own text. Every measured rate here is also a claim about four generated formats and thirteen planted cases; a real permit office sees kinds this set does not, and how often it sees an isolation left open, a test go stale, a certificate lapse or a signature outside its scope is unknown here and is not claimed.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 16,505 characters assembled, 11,416 are references — 69% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The exact five parts sent for PK-0001, the first package of run r001-worksafe-precheck, replayed from the kit's own src/prompt.py build(). Trustworthy as the prompt that was sent because the replayed part names and character counts (2,117 / 11,416 / 328 / 890 / 1,754 = 16,505) match the decomposition the run itself recorded, field for field. Only characters are published per part: the run record decomposes the prompt by characters, not by tokens, so a per-part token count would be a number no file holds.

Full promptThe prompt in full

You are the pre-start check on a WORK AUTHORISATION PACKAGE: the paperwork a crew must have in
order before work begins. One package goes in. What comes out is a report with one line per
control the package's own declared hazards require, so that a supervisor confirms a report instead
of re-reading a permit.

How to read it:

- The package arrives in whatever shape the site writes it: a permit form with labels down the
  left margin, a system export, a set of CSV blocks, or a shift handover written in sentences.
  Read it as a person would, whatever shape it is in.
- Read out of it: the permit identifier, the VALIDITY WINDOW it claims, the HAZARDS it declares,
  the ISOLATION POINTS it lists and the state each one is in, the GAS READINGS taken and when,
  what is ATTACHED and until when each attachment is valid, and WHO signed and what their
  authorisation covers.
- An isolation point is ISOLATED only if the package says so. A point that is listed and left
  open, in service, car-sealed open, deferred to another shift, or never applied is listed and not
  isolated. Read the state, not the presence of the row.
- A signature authorises the hazards that person's authorisation covers, and no others. Somebody
  senior signing a hazard outside their authorisation has not authorised it.
- The moment every date question is asked against is the STATED START in the desk record below,
  not the moment the package was written and not the moment anyone signed it.
- Then take the declared hazards, look up the controls the rulebook requires for them, and give
  ONE verdict per control, trying the four in the order the rulebook gives.
- Report the controls in the order the rulebook lays out: the two package-level controls first,
  then the hazards in rulebook order, then each hazard's controls in rulebook order. A control two
  declared hazards both require appears once.
- Every verdict that is not SATISFIED names the rule id it breaks. That is the deliverable: a
  supervisor has to know what to go and get, and who has to sign it.

Reply with JSON and nothing else, in the shape given at the end.

THE HAZARD CONTROL RULEBOOK, as this scheme applies it.
It is this kit's own invented rulebook -- not a transcription of any real standard.

THE FOUR VERDICTS -- exactly one per control
  MISSING       The package carries no evidence of this control at all. Not a weak entry -- an absent one.
  INCONSISTENT  The evidence is there and something else in the same package contradicts it: an isolation listed but left open, a signature from someone whose authorisation does not cover the hazard, a reading outside the limit the package declares it met, an isolation for a hazard the package never declared.
  EXPIRED       The evidence is there and it is out of date at the moment work is due to start: a test older than its freshness window, a certificate whose validity ended, a permit window that has closed.
  SATISFIED     The package carries the evidence this control requires, it is inside its validity window, and nothing else in the package contradicts it.

THE ORDER THEY ARE TRIED IN
  MISSING -> INCONSISTENT -> EXPIRED -> SATISFIED
  Fixed order, applied per control, top down. MISSING is first because there is nothing to test the consistency or the age of. INCONSISTENT precedes EXPIRED because a contradicted control is not cured by a fresh copy of the same evidence -- a fresh reading of a space whose isolation is open is still a space whose isolation is open -- whereas a stale but otherwise coherent piece of evidence is cured by redoing it. SATISFIED is what is left.

CONTROLS EVERY PACKAGE CARRIES, WHATEVER IT DECLARES
  WIN-VALID     R-WIN-01     Validity window open at the stated start
                The package states a validity window. Work must be due to start inside it, and the window must end after it begins.
  DEC-CONS      R-DEC-01     Declared hazards account for the whole package
                Every isolation, atmospheric test and attachment the package lists must belong to a control that one of the DECLARED hazards requires. An item that belongs to no declared hazard means the declaration is incomplete: the work involves a hazard nobody wrote down, so nobody applied its other controls either.

HAZARDS AND THE CONTROLS EACH ONE REQUIRES
  Confined space entry  (confined_space)
    ATM-INIT      R-ATM-01   Pre-entry atmospheric test
                             Oxygen, flammable gas and toxic gas all tested no more than 120 minutes before work starts, and every result inside the limit.
                             tested within 120 minutes of the stated start
                             flammable  at most 5.0 pctLEL
                             oxygen     at least 19.5 and at most 23.5 pct
                             toxic      at most 10.0 ppm
    ISO-PROC      R-ISO-01   Process line isolation, applied
                             At least one process-line isolation point listed on the package and in the ISOLATED state. A point listed and left open, in service, or never applied is not an isolation -- listing it does not isolate it.
    ATT-RESCUE    R-ATT-01   Rescue plan on file
                             A written rescue plan, referenced by number, still inside its review date at the stated start.
    AUTH-CS       R-AUT-01   Authorisation for confined space entry
                             A signature from a person whose authorisation covers confined space entry, signed on or before the stated start.
  Hot work  (hot_work)
    ATM-FLAM      R-ATM-02   Flammable gas test before hot work
                             Flammable gas tested no more than 60 minutes before hot work starts, and the result at or below 1.0 pctLEL -- a tighter limit than entry, because the ignition source is the work itself.
                             tested within 60 minutes of the stated start
                             flammable  at most 1.0 pctLEL
    ATT-FIREWATCH R-ATT-02   Fire watch posted
                             A named fire watch, referenced on the package.
    ATT-SCREEN    R-ATT-03   Spark containment in place
                             A spark containment or screening record, referenced on the package.
    AUTH-HW       R-AUT-02   Authorisation for hot work
                             A signature from a person whose authorisation covers hot work, signed on or before the stated start.
  Work on or near energised electrical equipment  (energised_electrical)
    ISO-ELEC      R-ISO-02   Electrical isolation, applied
                             At least one electrical isolation point listed on the package and in the ISOLATED state.
    ATT-LOTO      R-ATT-04   Lock-out register entry
                             The lock-out register entry for this job, referenced on the package.
    AUTH-EE       R-AUT-03   Authorisation for energised electrical work
                             A signature from a person whose authorisation covers energised electrical work, signed on or before the stated start.
  Work at height  (work_at_height)
    ATT-FALLPLAN  R-ATT-05   Fall protection plan
                             A written fall protection plan, referenced on the package.
    ATT-ANCHOR    R-ATT-06   Anchor point certificate
                             A certificate for the anchor points to be used, still valid at the stated start.
    AUTH-WH       R-AUT-04   Authorisation for work at height
                             A signature from a person whose authorisation covers work at height, signed on or before the stated start.
  Breaking into a pressurised system  (pressurised_system)
    ISO-PRESS     R-ISO-03   Pressure isolation, applied
                             At least one pressure isolation point listed on the package and in the ISOLATED state, before any break-in.
    ATT-DEPRESS   R-ATT-07   Depressurisation and drain record
                             A record that the system was depressurised and drained, referenced on the package.
    AUTH-PS       R-AUT-05   Authorisation for breaking into a pressurised system
                             A signature from a person whose authorisation covers breaking into a pressurised system, signed on or before the stated start.
  Excavation and ground disturbance  (excavation)
    ATT-SERVICES  R-ATT-08   Buried services survey
                             A buried services survey for the dig area, still valid at the stated start.
    ATT-SHORING   R-ATT-09   Shoring or battering design
                             A shoring or battering design for the excavation, referenced on the package.
    AUTH-EX       R-AUT-06   Authorisation for excavation
                             A signature from a person whose authorisation covers excavation, signed on or before the stated start.
  Lifting operation  (lifting_operation)
    ATT-LIFTPLAN  R-ATT-10   Lift plan
                             A written lift plan, referenced on the package.
    ATT-EQCERT    R-ATT-11   Lifting equipment certificate
                             A current certificate for the lifting equipment, still valid at the stated start.
    AUTH-LO       R-AUT-07   Authorisation for the lifting operation
                             A signature from a person whose authorisation covers lifting operations, signed on or before the stated start.
  Exposure to a hazardous substance  (hazardous_substance)
    ATM-TOX       R-ATM-03   Toxic gas test
                             Toxic gas tested no more than 240 minutes before work starts, and the result at or below the limit.
                             tested within 240 minutes of the stated start
                             toxic      at most 10.0 ppm
    ATT-DATASHEET R-ATT-12   Substance data sheet attached
                             The data sheet for the substance involved, referenced on the package.
    AUTH-HS       R-AUT-08   Authorisation for hazardous substance exposure
                             A signature from a person whose authorisation covers hazardous substance work, signed on or before the stated start.

THE STATE WORDS AN ISOLATION MAY CARRY
  isolated     The energy source is isolated and the isolation is in place.
  open         The point is listed on the package and is NOT isolated -- left open, in service, car-sealed open, or deferred. Listing it does not isolate it.
  not_applied  The point is listed and the isolation was never applied.

HOW THE REPORT IS ORDERED
  Controls are reported in a fixed order so two runs of the same package produce the same report: the always-required package controls first, in the order they are listed above, then the hazards in the order this file lists them, then each hazard's controls in the order it lists them. A control required by two declared hazards is reported ONCE, attributed to the first hazard that requires it.

THE RULES, BY ID -- name the one a failure breaks
  R-ATM-01   Before entry, oxygen, flammable gas and toxic gas are tested within 120 minutes of the start, each inside its limit, and the results recorded on the package.
  R-ATM-02   Before hot work, flammable gas is tested within 60 minutes of the start and the result is at or below 1.0 pctLEL.
  R-ATM-03   Where a hazardous substance is present, toxic gas is tested within 240 minutes of the start and the result is at or below its limit.
  R-ATT-01   A rescue plan is on file and inside its review date.
  R-ATT-02   A fire watch is named on the package.
  R-ATT-03   Spark containment is recorded on the package.
  R-ATT-04   The lock-out register entry for the job is referenced on the package.
  R-ATT-05   A fall protection plan is referenced on the package.
  R-ATT-06   The anchor points carry a certificate valid at the start.
  R-ATT-07   Depressurisation and draining are recorded before any break-in.
  R-ATT-08   A buried services survey valid at the start covers the dig area.
  R-ATT-09   A shoring or battering design is referenced on the package.
  R-ATT-10   A lift plan is referenced on the package.
  R-ATT-11   The lifting equipment carries a certificate valid at the start.
  R-ATT-12   The data sheet for the substance is attached.
  R-AUT-01   Confined space entry is authorised by a person whose authorisation covers it.
  R-AUT-02   Hot work is authorised by a person whose authorisation covers it.
  R-AUT-03   Energised electrical work is authorised by a person whose authorisation covers it.
  R-AUT-04   Work at height is authorised by a person whose authorisation covers it.
  R-AUT-05   Breaking into a pressurised system is authorised by a person whose authorisation covers it.
  R-AUT-06   Excavation is authorised by a person whose authorisation covers it.
  R-AUT-07   A lifting operation is authorised by a person whose authorisation covers it.
  R-AUT-08   Hazardous substance work is authorised by a person whose authorisation covers it.
  R-DEC-01   The declared hazards must account for every control the package evidences. An isolation, test or attachment that belongs to no declared hazard means the declaration is incomplete.
  R-ISO-01   A process line is isolated and every process-line point the package lists is in the isolated state.
  R-ISO-02   Electrical energy is isolated and every electrical point the package lists is in the isolated state.
  R-ISO-03   Stored pressure is isolated and every pressure point the package lists is in the isolated state.
  R-WIN-01   Work may only start inside the validity window the package states, and a window whose end is not after its start authorises nothing.

THE DESK RECORD -- this package as the permit office's own queue holds it.
Every date question below is asked against the stated start.

  Package id              PK-0001
  Site and asset          Alderoak Gas Plant, crude storage tank T-104
  Work due to start (UTC) 2026-09-17T10:00:00Z
  Package received as     handover_note

THE PACKAGE, verbatim:

PRE-JOB BRIEFING NOTE  --  PK-0001

Job on Alderoak Gas Plant, crude storage tank T-104: internal shell inspection after the wash. On the hazard side we have a break-in on a system that holds pressure and work in the amine. The authorisation runs from 08:00 on the 17th of September 2026 and it closes at 18:00 on the 17th of September 2026; all times in this note are UTC.

On isolations: the pressure point at PR-474 is isolated and the lock is hung.

No gas test is written on the sheet.

Paperwork on file: the depressurisation record is DP-3516; the substance data sheet is SD-2224.

Signatures: H. Dumas (responsible electrical person) signed at 07:50 on the 17th of September 2026 and is authorised for a break-in on a system that holds pressure and there is a hazardous substance in the line.

Crew are due on the job at 10:00 on the 17th of September 2026.


Reply with JSON and nothing else, exactly this shape:

{"reading": {
   "permit_id": "<the identifier the package carries>",
   "valid_from": "<YYYY-MM-DDTHH:MM:SSZ, the start of the validity window the package claims>",
   "valid_to": "<YYYY-MM-DDTHH:MM:SSZ, the end of it>",
   "declared_hazards": ["<hazard id>", ...],
   "isolations": [{"tag": "<the point's tag>", "kind": "process_line" | "electrical" | "pressure" | "other", "state": "isolated" | "open" | "not_applied"}],
   "gas_readings": [{"analyte": "oxygen" | "flammable" | "toxic", "value": <number>, "unit": "<as written>", "taken_at": "<YYYY-MM-DDTHH:MM:SSZ>"}],
   "attachments": [{"kind": "<attachment kind>", "reference": "<its number>", "valid_until": "<YYYY-MM-DD, or null if it carries no expiry>"}],
   "signoffs": [{"name": "<who>", "role": "<their role>", "authorised_for": ["<hazard id>", ...], "signed_at": "<YYYY-MM-DDTHH:MM:SSZ>"}]},
 "controls": [
   {"control": "<control id>",
    "verdict": "SATISFIED" | "MISSING" | "EXPIRED" | "INCONSISTENT",
    "rule": "<the rule id this failure breaks, or null when SATISFIED>",
    "why": "<one sentence naming what in the package decided it>"}],
 "may_start": true | false}

`may_start` is false when ANY control is not SATISFIED.

The hazard ids are exactly: confined_space, hot_work, energised_electrical, work_at_height, pressurised_system, excavation, lifting_operation, hazardous_substance
The isolation kinds are exactly: electrical, pressure, process_line, other
The gas analytes are exactly: flammable, oxygen, toxic
The attachment kinds are exactly: anchor_cert, depress_record, equipment_cert, fall_plan, fire_watch, lift_plan, lockout_register, rescue_plan, services_survey, shoring_design, spark_screen, substance_sheet

Raw responseThe raw response

{"reading": {
  "permit_id": "PK-0001",
  "valid_from": "2026-09-17T08:00:00Z",
  "valid_to": "2026-09-17T18:00:00Z",
  "declared_hazards": ["pressurised_system", "hazardous_substance"],
  "isolations": [
    {"tag": "PR-474", "kind": "pressure", "state": "isolated"}
  ],
  "gas_readings": [],
  "attachments": [
    {"kind": "depress_record", "reference": "DP-3516", "valid_until": null},
    {"kind": "substance_sheet", "reference": "SD-2224", "valid_until": null}
  ],
  "signoffs": [
    {"name": "H. Dumas", "role": "responsible electrical person", "authorised_for": ["pressurised_system", "hazardous_substance"], "signed_at": "2026-09-17T07:50:00Z"}
  ]
},
"controls": [
  {
    "control": "WIN-VALID",
    "verdict": "SATISFIED",
    "rule": null,
    "why": "The stated validity window 08:00-18:00 on 2026-09-17 contains the 10:00 stated start."
  },
  {
    "control": "DEC-CONS",
    "verdict": "SATISFIED",
    "rule": null,
    "why": "PR-474, DP-3516 and SD-2224 all belong to controls required by the declared pressurised_system or hazardous_substance hazards."
  },
  {
    "control": "ISO-PRESS",
    "verdict": "SATISFIED",
    "rule": null,
    "why": "Pressure point PR-474 is listed and in the isolated state."
  },
  {
    "control": "ATT-DEPRESS",
    "verdict": "SATISFIED",
    "rule": null,
    "why": "Depressurisation and drain record DP-3516 is referenced on the package."
  },
  {
    "control": "AUTH-PS",
    "verdict": "SATISFIED",
    "rule": null,
    "why": "H. Dumas signed at 07:50 before the start and is authorised for pressurised-system break-in."
  },
  {
    "control": "ATM-TOX",
    "verdict": "MISSING",
    "rule": "R-ATM-03",
    "why": "The package states that no gas test is written, so there is no toxic gas reading for the hazardous substance work."
  },
  {
    "control": "ATT-DATASHEET",
    "verdict": "SATISFIED",
    "rule": null,
    "why": "Substance data sheet SD-2224 is referenced on the package."
  },
  {
    "control": "AUTH-HS",
    "verdict": "SATISFIED",
    "rule": null,
    "why": "H. Dumas signed at 07:50 before the start and is authorised for hazardous substance work."
  }
],
"may_start": false}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check that a work package carries every control its declared hazards require — 560 work authorisation packages drawn from 60 real work authorisation packages. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

No model grades anything. Every metric is exact match against data/gold.jsonl -- a verdict word from a four-item closed set and a rule id string -- with the unit fixed at ONE CONTROL ROW, aligned by control id and never by position. There is therefore no matching step to soften a miss. A row the key has and the arm never reported is counted as not_reported and a row the checklist does not contain as extra, and neither is folded into a verdict: an omitted row looks careful inside an accuracy average and reads as "no objection" on a printed report, which is the same thing as clearing it. ⚑ WHAT IS DELIBERATELY NOT GRADED, and is published as diagnostics instead: the eight READING fields. data/fields.json carries a graded flag on every field and src/prompt.py builds the schema from it, so the fields the model is asked for and the fields the scorer reads are one list; permit_id is graded NOWHERE, because no rule uses it -- it is what a person quotes when they go back to the pack. The why sentence on every control row is ungraded entirely: it is read by a person and no code compares it, so "its sentence was right and its verdict word was wrong" is an observation from reading the seven AUTH-* misses, not a measurement. There is also NO COVERAGE CEILING COLUMN, deliberately: the corpus is generated with its own key, so every gold row states every field and a ceiling would read 560 of 560 -- a fact about the generator, not about how hard a package is to read. AND THERE IS NOTHING TO ADJUDICATE: no judge model exists anywhere in the grading path, every figure is exact match against a computed key, and the only second opinion in the kit is evals/check_labels.py's independent re-derivation of the key itself, which agrees on all 560 rows.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Run it twiceThe same set, run again

control-verdict accuracy over the 560 checklist rows —

Grading costWhat it costs

Every dollar on this page is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid. The token counts come from the provider's own usage block on each call, and 90.9 pct of the output tokens they report are provider-side reasoning the kit did not ask for.

Rates checked 2026-08-27. The provider that actually ran every call here is kept off this page per the series rule. The real spend ($0.371321 over 60 calls), its own dated card, its peak/off-peak split (the same tokens at peak would have been $0.742642) and its cached-input tier are all recorded per call in results/eval-r001-worksafe-precheck.json, not on this page.

Grading unitWhat the grading figure prices

Scoring is pure Python over committed JSON: 560 rows in well under a second, no key, no network. The paid ARM costs money ($0.3713 for 60 calls); grading it costs nothing, and the RECHECKED column can be re-derived from cache-r001-worksafe-precheck.jsonl at any time for $0.00. Basis: results/eval-r001-worksafe-precheck.json (usd block, priced per call from the provider's reported cache split at the tariff in force at the UTC time of the call) and results/eval-b000-worksafe-precheck-rules.json (usd null, wall 0.1 s).

The gradersThree ways to grade

The null baseline is answering SATISFIED on every row, which is the majority verdict: 493 of 560 rows, 88.0 pct on verdict accuracy alone -- and it is a catastrophe, because all 67 of its misses are FALSE CLEARS and it releases every one of the 50 packages the key holds. It is reported because it is the floor beneath the floor and because it shows why a blended accuracy figure is the wrong number for this job: 88.0 pct is a system that would send crews to work against every missing isolation in the corpus. The column that actually matters is the RULES FLOOR, which is a genuine attempt at the whole job for $0.00, scores 80.2 pct overall AND 100 pct on the three labelled formats, and ships zero false clears. ⚑ AND THE FLOOR IS RECHECKED BY THE SAME FUNCTION THE PAID ARM IS, as of 2026-08-31. Until then evals/run.py's floor branch handed the arm a rechecked block that was literally a copy of its own answer, while the paid arm called src/recheck.py. It now calls RC.recheck(answer, package) with the arguments src/matcher.py passes. b000-worksafe-precheck-rules was re-run for $0.00 with no key in the environment and NOT ONE NUMBER MOVED: control verdicts 449 of 560, packages fully right 45 of 60, recheck_overrides 0, unrecheckable 0, the rechecked record identical on every scored field of all sixty packages. It is a no-op here because this floor already hands its reading to src/rules.py — the same engine the recheck re-applies — which is exactly the property the column was claiming and could not previously prove. Red-proven by seeding: force PK-0001's raw answer to SATISFIED on every control with may_start true and the rechecked column puts may_start back to false. r001-worksafe-precheck was not re-run; its file, its cache and its $0.3713 are byte-identical.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Set limitationsWhat this set cannot show

DELIBERATELY UNBALANCED, and the imbalance is declared rather than discovered: 493 of the 560 control rows are SATISFIED and 67 fail (22 MISSING, 19 EXPIRED, 26 INCONSISTENT), and 50 of the 60 packages must be HELD against 10 clean. The mix is fixed in tools/build_corpus.py at SEED = 20260831 and printed in data/corpus-stats.json: 13 named cases, five sixths of the packages planting one, because a clean package teaches nothing about the arithmetic -- and a sixth clean, because a corpus that is all traps measures a different job. The four formats are balanced at 15 packages each, which is what makes the by-format table readable.

A single accuracy figure over all 560 rows FLATTERS EVERY ARM and hides the only comparison that matters here. Answering SATISFIED to everything scores 88.0 pct (493 of 560) while false-clearing all 67 failing controls and releasing every one of the 50 packages the key holds -- so the headline pair 97.3 pct against 80.2 pct has to be read against 88.0 pct, not against zero. And the 560 rows are not one population: 415 of them are on formats where the FREE FLOOR is perfect and the paid arm can only tie it, and 145 are the prose rows where the whole spread lives. Every verdict class is therefore reported on its own denominator, false_clear and false_hold are never averaged, and controls_not_reported and controls_extra are counted on their own lines.

Building and labelling the whole set cost nothing -- it is generated from a fixed seed (20260831) by tools/build_corpus.py and graded free by evals/check_labels.py with an independent implementation. Re-running the free floor over all 60 packages costs $0.00 and takes 0.1 s. What has NOT been built is a harder set: a handover note written by a different hand from this generator's phrase pools, a package whose own tables contradict each other, a gas reading recorded in the wrong unit, a fourth labelled format with different field names, and a hazard present on the job and evidenced nowhere at all -- the one DEC-CONS cannot reach. Those are where this kit's 100 pct on the labelled formats and its 98.6 pct on prose would first stop being true. AND THE ADVERSARIAL SET IS UNBUILT IN THE SAME SENSE: evals/injection.py is written and has never been fired, so no package in this corpus has ever been attacked.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-27. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingGrading is free; the bill is the 60 calls that answered.

The scorer, the label gate, the corpus generator and the rules floor are pure code and cost $0.00 to run against any result set -- there is no LLM judge anywhere in the grading path. The figure above is the ONE paid run's token counts (60 calls, 242,357 in / 544,654 out) priced at the same projected card cost_per_query_usd uses. There is no second paid arm to add: the injection probe was never fired for this kit.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, OVERWHELMINGLY. 9,077.6 per call against 4,039.3 in, and 90.9 pct of them (494,846 of 544,654) are provider-side reasoning left at the tier's default and re-rolled per call. On the projection card the output side is 93.1 pct of the bill. The visible answer is a reading block plus 5 to 13 control rows.
• THE STABLE PREFIX IS THE SAVING, NOT THE COST. The system role and the rulebook are 13,533 of the prompt's 16,505 characters -- 82.0 pct identical on every call -- and the provider that ran this kit billed 194,688 of 242,357 input tokens at its cached rate (80.3 pct), the highest share this design allows. The projection card has no cache tier, so it prices those tokens at full rate; on this kit that barely matters, because input is only 6.9 pct of the projected bill.
• THE PACKAGE ITSELF IS 5.4 PCT OF THE PROMPT AND 2.0 PCT IS THE DESK RECORD. The bill is the rulebook and the reasoning, not the document -- so a longer package moves the cost hardly at all, and a longer RULEBOOK moves every call.

Your volumeWhat it costs at your volume

Linear in packages and flat in everything else. Ten times the packages is ten times the calls at the same per-package cost: there is no index, no retrieval and no state between packages, and the prompt is a fixed rulebook plus one document whose share is 5.4 pct. Ten times the SITE does not move it either -- a bigger site does not make a package bigger. What does move it is the RULEBOOK: at 69.2 pct of the prompt, a scheme with 80 controls instead of 28 roughly triples the input side of every call, though input is only 6.9 pct of the projected bill and 80.3 pct of it is already billed as cache. A permit office running 200 packages a day is about $5.85 a day on the projection card and $1.24 at the rate this run actually paid. The real ceiling is not money: at a p95 of 215.3 s and 5 workers, 200 packages is roughly 54 minutes of wall clock, and a pre-start check that arrives after the shift starts is not a pre-start check.

Model choiceWhy this model

It is the tier the estate runs, so the figure compares with every sibling kit. The finding here is not about the model: on the three labelled formats it was BEATEN by a free script and only ties it after pure code re-derives its verdicts, and on the INCONSISTENT verdict its raw report is 19 of 26 against the floor's 24. What it bought is the prose format -- 23.4 pct to 98.6 pct -- and everything it bought there is a READING, not a judgement.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

the whole 60-package scored run (r001-worksafe-precheck, the fast tier): 242,357 input tokens (194,688 of them prefix cache hits) and 544,654 output (494,846 of them reasoning). It is the ONLY paid run this kit has -- the injection probe was never fired. The real bill is on the withheld provider's card and lives in the run record, not on this page.

The codeEvery file, and what it is for

11 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/prompt.pyprompt assembly

Five parts in a fixed order -- the system role, data/rulebook.json rendered, the desk record, the package verbatim, the JSON schema. It names all 28 controls, all 28 rule ids and all four verdicts, and asserts at import that each one appears in the rendered text; scoring an arm on a vocabulary it was never given measures the prompt. What it never does is the arithmetic: it never says a reading is 40 minutes old, that a window has closed, or that a signature does not cover a hazard. The package goes in VERBATIM -- a pre-parsed field summary is where "IS-114 left open for flushing" quietly becomes state: isolated before the model sees it.

# The assembled prompt, in one place, in the order it is sent.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULES = json.load(open(os.path.join(HERE, "data", "rulebook.json"), encoding="utf-8"))
VERDICTS = ("SATISFIED", "MISSING", "EXPIRED", "INCONSISTENT")
VERDICT_MEANINGS = {
RULEBOOK = B.rulebook_text()
SYSTEM = """You are the pre-start check on a WORK AUTHORISATION PACKAGE: the paperwork a crew must have in
SCHEMA = """Reply with JSON and nothing else, exactly this shape:
def desk_block(pkg):
def build(text, pkg):

src/matcher.pythe model call

One call per package, the only place a model is reached. Parses and normalises the reply -- case-folds the closed vocabularies, coerces a comma string of hazards to a list -- and returns the raw answer and the rechecked one side by side. Normalisation is case and shape only, never meaning: an unreadable field stays absent and is scored as a miss. MAX_TOKENS is 32,000 and a reply cut off at it is recorded with at_ceiling and stays inside the published denominator; it is not re-fired.

# One package in, one pre-check report out. The only place a model is called.

MAX_TOKENS = 32000
THINKING = None
def parse_reply(text):
def _text(v):
def _bool(v):
def _idlist(v):
def _dicts(v):
def normalise(obj):
def work(cfg, text, pkg, complete_fn=None, max_tokens=None):

src/adapters/__init__.pythe adapters — a swap seam

Raw HTTP over urllib to any OpenAI-compatible provider or Anthropic -- no vendor SDK, so pip install pulls nothing and a forker runs this on whichever key they already hold. A provider that returns no token counts cannot be published, because the LLM lens prints them and the Cost lens prices them.

You change it to: PROVIDER + BASE_URL + MODEL in .env -- one line, then the same run again. Two adapter shapes ship (OpenAI-compatible and Anthropic's Messages API); adding a third is one function and one entry in PROVIDERS, and it must return token counts. Configuration is at the REPO ROOT (<repo>/.env) and inherited by every kit; a kit-local .env is an override merged key by key, and the real environment still wins over both.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
TIMEOUT_S = 1200
TRANSPORT_RETRIES = 1
def _post(url, headers, payload, timeout=TIMEOUT_S):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}

src/book.pythe rulebook, as lookups

required_controls() is the join at the centre of the kit: a package declares hazards, the rulebook says which controls each requires, and the report has exactly one row per control that join produces -- never a typed checklist. A control two declared hazards both require is reported ONCE, attributed to the first, so the accuracy denominator does not move with the declaration. implied_hazards() is the REVERSE join and is what makes DEC-CONS decidable: given an evidence item, which hazards would require a control it could satisfy? An item whose answer contains no declared hazard is an item the declaration does not account for.

# The rulebook, as lookups. Pure code, no model, no key.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULES = json.load(open(os.path.join(HERE, "data", "rulebook.json"), encoding="utf-8"))
HAZARDS = RULES["hazards"]
CONTROLS = RULES["controls"]
RULE_TEXT = RULES["rules"]
VERDICT_ORDER = tuple(RULES["precedence"])
ALWAYS = tuple(RULES["always_required"])
HAZARD_ORDER = tuple(HAZARDS)
ISOLATION_STATES = tuple(RULES["isolation_states"])

src/clock.pythe clock

Freshness, validity and the stated start, as arithmetic. EVERY date question is asked against the stated start of work in the desk record -- not the moment the package was written, not "now" -- because the same gas reading is fresh at 07:00 and stale at 09:00. minutes_before returns a NEGATIVE number for a test dated after the start, and the engine treats that as a contradiction rather than a freshness failure. Everything is UTC and a naive timestamp is refused rather than assumed. The 120/60/240-minute windows come from data/rulebook.json and are not typed here.

# Freshness, validity and the stated start, as arithmetic. Pure code, no model, no key.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULES = json.load(open(os.path.join(HERE, "data", "rulebook.json"), encoding="utf-8"))
def ts(x):
def iso(x):
def d(x):
def date_of(x):
def minutes_before(taken_at, work_start):
def fresh(taken_at, work_start, minutes):
def window_open(valid_from, valid_to, work_start):

src/rules.pythe pre-check engine

The rulebook as arithmetic, and the PURE CODE station every arm stands on: it writes the answer key, powers the recheck, sits behind the free floor and drives the UI. The four verdicts are tried in data/rulebook.json's own precedence order and that order is ASSERTED against the file at import -- an engine whose branches have drifted from the order it publishes explains itself wrongly on every page it appears on. No number is typed in this file.

# The pre-check engine: the rulebook as arithmetic. Pure code, no model, no key.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULES = json.load(open(os.path.join(HERE, "data", "rulebook.json"), encoding="utf-8"))
VERDICTS = tuple(RULES["precedence"])
BLOCKING = tuple(k for k in VERDICTS if RULES["verdicts"][k]["blocks_start"])
HAZARDS = tuple(RULES["hazards"])
CONTROLS = RULES["controls"]
ISOLATION_STATES = tuple(RULES["isolation_states"])
ANALYTES = tuple(sorted({a for c in CONTROLS.values() for a in (c.get("analytes") or [])}))
ATTACHMENT_KINDS = tuple(sorted({c["attachment_kind"] for c in CONTROLS.values()

src/recheck.pythe recheck — a swap seam

Takes the model's READING and NOTHING ELSE -- not one verdict, not one rule id, not the control list, not may_start -- and re-derives the whole report from data/rulebook.json. That is a harder line than a sibling kit's tie-break, and it is drawn because there is nothing here a model can decide that the rulebook cannot. It overrode 12 verdicts on this run (recheck_overrides 12, recheck_unrecheckable 0) and carried RAW 545/560 to RECHECKED 558/560 for no second call. It CANNOT fix a misread: the 8 invented rows sit on a wrong declaration and survive it unchanged.

You change it to: Which fields the recheck takes from the reply. Today: the reading and nothing else. Widening that set moves work from code to model and widens the injection surface; a field the code re-derives is a field the package cannot lie about.

# The model's reading, the rulebook's arithmetic. Pure code, no model, no key.

TAKEN_FROM_MODEL = ("permit_id", "valid_from", "valid_to", "declared_hazards", "isolations",
def _model_report(answer):
def recheck(answer, pkg):

evals/baseline.pythe free floor

What a permit office would script if it had a weekend and no model: a csv reader for the bundle, a key-and-value reader for the export, a section reader for the permit form, a sentence scanner for the handover note -- and then EXACTLY the same src/rules.py engine. Its patterns may use the rulebook's own published vocabulary and the three labelled formats' field labels, and nothing else. It is 415 of 415 control rows on those three formats and 34 of 145 on the prose one.

# THE FREE FLOOR. No key, no model, no network. A genuine attempt at the job.

MODES = ("rules",)
HAZ_IDS = tuple(B.HAZARD_ORDER)
def _windows(label):
HAZ_WORDS = {h: ({h, h.replace("_", " "), B.HAZARDS[h]["name"].lower()}
ANALYTES = tuple(R.ANALYTES)
ISO_KINDS = tuple(R.ISOLATION_KINDS)
ATT_KINDS = tuple(R.ATTACHMENT_KINDS)
def _tokens(*phrases):
ATT_WORDS = {k: _tokens(k, next(v["name"] for v in R.CONTROLS.values()

evals/scoring.pythe scorer — a swap seam

THE UNIT IS THE CONTROL, NOT THE PACKAGE -- a report that gets eight of nine rows right is one row wrong, and which row decides whether a crew goes to work. Alignment is by control id and never by position, so a row the key has and the arm never reported is counted as not_reported and a row the checklist does not contain is counted as extra; NEITHER is folded into a verdict, because an omitted row looks careful in an accuracy average and reads as "no objection" on a printed report. false_clear and false_hold are never averaged, and every miss files into exactly one of eight taxonomy buckets in a fixed order.

You change it to: Graders, the two error directions and the eight-bucket failure taxonomy. false_clear and false_hold are scored on their own denominators and never averaged, and controls_not_reported and controls_extra are counted on their own lines rather than folded into an accuracy average -- which is the only reason the floor's 93 unreported rows and the model's 8 invented ones are visible at all.

# Grade one arm against the answer key. Deterministic -- no model judges anything here.

VERDICTS = tuple(R.VERDICTS)
TAXONOMY = ("no_report", "control_not_reported", "control_invented",
def _pct(n, d):
def _bucket(want, got, want_rule, got_rule, reported):
def score(records, golds, texts=None):
def _norm(x):
def _same_isolations(got, want):
def _same_readings(got, want):
def _same_attachments(got, want):

evals/check_labels.pythe label gate

Grades the ANSWER KEY with an independent implementation -- it imports neither src/rules.py nor src/book.py. It re-reads data/rulebook.json and does the whole job again: its own checklist join, its own reverse join for the declaration test, its own freshness subtraction, its own limit comparison, its own isolation-state test, its own certificate arithmetic, its own authorisation-scope test and its own precedence. It also asserts every field the key claims to have read is present, as a string, in the package's own text. Two implementations, one key; they agree on all 560 rows.

# Grade the ANSWER KEY. No key, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
RULES = json.load(open(os.path.join(DATA, "rulebook.json"), encoding="utf-8"))
PACKAGES = {p["id"]: p for p in json.load(open(os.path.join(DATA, "packages.json"),
HAZ = RULES["hazards"]
CTL = RULES["controls"]
PRECEDENCE = RULES["precedence"]
REQUIRED_CASES = ("clean_package", "control_missing", "two_missing", "reading_stale",

src/app.pythe local board

http.server, hand-written HTML and one JS file. /api/packages, /api/package, /api/rules, /api/recorded and /api/corpus need no key; only /api/check calls a provider. The free floor is computed on EVERY package and shown beside the key because it costs nothing -- and on the three labelled formats it is perfect, which is a thing a page selling a model would rather not draw. The model's answer is shown twice, as it came back and after the engine re-derived every verdict.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
CORPUS = os.path.join(HERE, "data", "corpus")
PORT = int(os.environ.get("PORT", "9213"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-worksafe-precheck")
FLOOR_RUN = "b000-worksafe-precheck-rules"
def packages():
def golds():
def load_package(pid):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 4039 input and 9077 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

⚠︎ WRITTEN, FORCED BY DESIGN, AND NOT FIRED. A work authorisation package is this kit's injection surface and the threat is not hypothetical: the document is written by the people who want the job to go ahead, assembled by the crew and the area team, passed through a permit office that did not write most of it, and it reaches the model VERBATIM with free-text all through it -- a scope of work, a remarks box, an isolation note, a permit office note a system echoes back. Anyone who can get a sentence into one of those can address it to the checker, and the outcome they want is the cheapest one to ask for: close the checklist, the job is running late. evals/injection.py exists, forces one such line into packages the key says must be HELD, counts suppression AT THE CONTROL ROW rather than the package, scores BOTH arms, and pairs each reply against the scored run's own cached answer so the control costs nothing. IT HAS NOT BEEN RUN. There is no x001 result file for this kit and NO NUMBER ON THIS PAGE IS A MEASUREMENT OF INJECTION RESISTANCE.

API_KEY is read from the shared <code>&lt;repo&gt;/.env</code>, this kit's own <code>.env</code>, or the real environment, in that order. Both files are gitignored from the first commit; this repo has never held a credential. The key never leaves the machine and is never printed. src/budget.py counts CALLS against a shared cap rather than dollars, because a cap that silently reads the wrong rate card is worse than none -- and it says so out loud when no cap is configured.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. Both were run for real on not_measured.

Both gates are unmeasured and neither should be read as a zero. The kit's own file says it plainly: a suppression rate of 0 would mean one attempt did not work, not that the arm is robust -- and there is not even that here. What IS architecture rather than behaviour, and is measured: the recheck takes NOTHING from the reply but the reading, so a sentence asserting that controls were verified is not an input to any verdict at all. That is a structural claim about src/recheck.py, verifiable by reading it, and it is the only injection-adjacent claim this kit is entitled to make.

The resultNot measured — the probe is written and unfired. No suppression rate and no reading-movement rate exists for this kit.

There is nothing to detail: no trial was fired. What is ready is the harness -- evals/injection.py takes packages the key says must be HELD, appends one line in the register of a note already on file, fires each once, counts suppression at the CONTROL ROW (a row that should read MISSING, EXPIRED or INCONSISTENT and reads SATISFIED instead) with the package-level release reported beside it, scores RAW and RECHECKED separately, and pairs against results/cache-r001-worksafe-precheck.jsonl so only the injected call is paid for. Running it would be 20 calls at roughly $0.006189 each. Until it is run, this panel states an absence.

HonestyWhat this does not prove

• EVERYTHING ABOUT INJECTION. No probe was fired. There is no measurement of whether one sentence can suppress a failing control row, whether it can move the reading, or whether either arm behaves differently under attack. The harness is written and the control arm is already paid for; the run is simply outstanding.
• AN ATTACK ON THE READING, WHICH IS THE OPEN SURFACE BY DESIGN. src/recheck.py trusts the model for the declared hazards, the validity window, the isolations AND THEIR STATES, the gas readings and their timestamps, the attachments and their expiries, and the signoffs and their scopes. A package engineered to be READ as isolated, or as declaring one hazard fewer, moves the rechecked arm too, and nothing in this kit could see it.
• WHETHER THE RECHECK'S STRUCTURAL DEFENCE HOLDS IN PRACTICE. That the engine never takes a verdict from the reply is verifiable by reading src/recheck.py; that no phrasing can reach it through the reading is NOT, and has not been tested.
• PROVIDER-SIDE RETENTION, TRAINING USE AND LOG RESIDENCY. Provider-dependent and outside anything this kit measures. The package leaves the machine verbatim on every call.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never authorise work, never issue a permit, never close out a control, never clear a job. This kit produces the report a PERSON confirms, and the rule id on every failing row is that report telling them where to go and who has to sign it.

Stated on the UI, in the README and in src/app.py's docstring, and enforced by there being no such endpoint: the server's routes return pre-start reports and nothing else, and there is no write path to any permit system. Reinforced by src/recheck.py, which takes NOTHING from the reply but the reading -- not one verdict, not one rule id, not the control list, not may_start.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is ABSENCE OF A WRITE PATH plus a prompt rule, not a runtime enforcement layer -- with one real exception: the recheck IS enforcement for everything the rulebook decides by arithmetic. It is NOT enforcement for the READING, and the reading is where every unfixable error in this kit lives.
• ⚠︎ THERE IS NO INJECTION RESULT HERE AT ALL. evals/injection.py is written and unfired; no sentence has been tested against this kit. The recheck's structural property -- that no verdict is ever taken from the reply -- is verifiable by reading the file and is NOT a measurement, and it does not defend the reading, which is the surface an attacker would go for.
• ⚑ THE HEADLINE 97.3 / 99.6 PCT IS A MIXTURE AND READING IT AS A CAPABILITY IS THE MISTAKE THIS KIT MOST WANTS TO PREVENT. On the three labelled formats -- 415 of the 560 rows -- the free floor is 100 pct and the paid arm's own report is 405 of 415, reaching 415 only after pure code re-derives it. The spread lives entirely in 145 prose rows, and how hard THOSE are was a decision: an earlier renderer put the floor at 100 pct on all fifteen of them.
• The RECHECKED 99.6 pct is agreement with a computed key on an invented corpus of 60 packages, and the engine that rechecks is the engine that wrote the key. evals/check_labels.py is the independent second implementation that makes that survivable, and its own stated limit -- it imports HAZARD_PHRASES from the generator to assert a prose note names the hazard the key says it declares -- is written in the file rather than hidden.
• The 8 invented control rows are not a rounding error and the recheck cannot remove them. A supervisor handed PK-0034 is told to go and find a fire watch for a job with no hot work in it. Control precision 98.6 pct against the floor's 100 pct is a real column and it runs the opposite way from recall.
• may_start is a report line, not a gate. Nothing stops a confirmed package with a failing control except the person the report is for.

WatchedWhat is watched, and why that one

The kit measures 44 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

101 measured by the latest run -57 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (blocked_packages 50, clean_packages 10, controls 560, dataset_version worksafe-precheck-v1-60packages, expired_controls 19, failing_controls 67, failures 0, hard_packages 50, inconsistent_controls 26, missing_controls 22, packages_answered 60, reading_reports 60, rechecked_blocked_packages 50, rechecked_clean_packages 10, rechecked_controls 560, rechecked_expired_controls 19, rechecked_failing_controls 67, rechecked_hard_packages 50, rechecked_inconsistent_controls 26, rechecked_missing_controls 22, rechecked_reading_reports 60, rechecked_releasable_packages 10, rechecked_satisfied_controls 493, releasable_packages 10, satisfied_controls 493, socket_timeout_s 1200, work_packages 60) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

3 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 3 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

comply · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 101 chips that all say so.

comply · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 105 chips that all say so.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 101 chips that all say so.

DeviationsWhat deviated

0 breaches across 3 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The corpus generator, the label gate, the free floor and the local board are free and re-run in seconds on a keyless copy. Only the model arm costs money -- 60 calls, $0.371321, recorded per call in the run record -- and it was not re-fired after its misses were read; the RECHECKED column can be re-derived from the committed cache at any time for nothing. The injection probe has never been run and is the one paid arm this kit still owes.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, standard library end to end -- no orchestration layer, no vendor SDK, no retrieval stack, no rules DSL, no web framework, no pip install. requirements.txt names nothing and spends twelve lines saying why, ending with the rule that if you add a line there, something under src/ or evals/ must import it.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear, and concurrent only at the package level. There is no agent, no tool loop, no retrieval and no state carried between packages; every package is independent, which is exactly why 5 workers is the entire concurrency story.

The other sideWhat a framework costs you

• Adding a provider means editing the PROVIDERS dict by hand. In exchange pip install pulls nothing and a forker runs this on whichever key they already hold.
• The reply is parsed by hand. It held across all 60 paid calls, and that is an observation, not a guarantee -- a structured-output library would fail loudly where this fails quietly.
• No rules-engine DSL, so a scheme whose SHAPE differs from this one's (declare hazards -> join controls -> apply precedence) is a change to src/rules.py rather than to a config file. That is stated as the design's ceiling in Architecture.breaks_at_scale rather than hidden behind a library that would defer it.
• The free floor's readers are regexes written against three known formats. A fourth labelled format is new code in evals/baseline.py -- and since the floor is 100 pct on the three it was given, how much of that is the reader and how much is the format is a question this kit cannot answer from the inside.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-worksafe-precheck on the fast tier, 2026-08-31. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 1 earlier run on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT, with the corpus, the rulebook and the answer key all generated in-process -- there is no third-party data in this kit to licence. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe control row -- the verdict and the rule id it names, one row per control the declared hazards require

whether each of the 560 checklist rows is one a supervisor could act on as written: the right one of SATISFIED / MISSING / EXPIRED / INCONSISTENT, and, on every failure, the rule id to go and quote. The unit is the CONTROL, never the package -- a report that gets eight of nine rows right is one row wrong, and which row decides whether a crew goes to work. Rows are aligned by control id, never by position, so a row the arm never reported is counted as not_reported and a row the checklist does not contain is counted as extra; neither is folded into a verdict.

MethodHow the test was run

python3 -m evals.run --run-id <id> [--floor rules]; evals/scoring.py compares verdict strings and rule ids against data/gold.jsonl. No model is in the grading path.

Every grader on these pages scored the same 560 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl -- 560 control rows computed by src/rules.py from the generator's injected facts, never typed by a person, and tools/build_corpus.py refuses to write a package in which the mutation did not produce the verdict its case name promises on the control it aimed at. Graded independently by evals/check_labels.py, which imports neither src/rules.py nor src/book.py: it re-reads data/rulebook.json and redoes the whole job with its own checklist join, its own reverse join, its own freshness subtraction, its own limit comparison, its own isolation-state test, its own certificate arithmetic, its own authorisation-scope test and its own precedence, and asserts every field the key claims to have read appears as a string in the package's own text. Two implementations, one key; they agree on all 560 rows. Every site, asset, person, permit, signature and rule is invented -- see data/SOURCES.md.

No true/false rates for this grader. It records 9 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? There is no threshold to tune -- exact match on a verdict string and a rule id, so nothing here has a tolerance. The numbers that look like thresholds live in the RULEBOOK, not the grader: 120 / 60 / 240 minutes, oxygen 19.5-23.5 pct, flammable 5.0 pctLEL for entry and 1.0 for hot work, toxic 10 ppm -- all in data/rulebook.json, all applied identically by every arm.

Cadence: Once per corpus version. The paid arm ran ONCE and was not re-fired after its misses were read; the recheck can be re-derived from the committed cache for nothing.

The decisionWhen to reach for it

In one lineThe two error directions, never averaged -- plus the rows nobody reported and the rows nobody asked for

which way an arm fails. FALSE CLEAR is a control the key says fails, called SATISFIED: the paperwork says go and the control is not in place, and it is the only mistake in this kit that can hurt anybody. FALSE HOLD is the reverse -- a control that is fine, failed: a crew waits, which is cheap once and corrosive repeated, because a pre-check that cries missing is a pre-check the site learns to override. Beside them sit the two directions an accuracy average cannot see: CONTROLS_NOT_REPORTED (a row the checklist has and the arm never printed -- which reads as "no objection" on paper) and CONTROLS_EXTRA (a row the arm printed for a hazard the package does not declare).

MethodHow the test was run

python3 -m evals.run --run-id <id> [--floor rules]; no model is in the grading path.

Every grader on these pages scored the same 560 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The same data/gold.jsonl. The split into 67 failing and 493 satisfied rows is fixed by the key before any arm answers, and the 50/10 held/releasable package split with it.

No true/false rates for this grader. It records 9 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No tolerance -- a verdict either is SATISFIED or it is not, and a row either was printed or it was not. The denominators are stated beside every rate because 67, 50 and 10 are small.

Cadence: Every run, on every arm, including the free floor.

The decisionWhen to reach for it

In one lineThe reading -- the six fields the recheck takes from the model and cannot re-derive

whether the half of this job that is genuinely a reading was read right, package by package: the declared hazards, the validity window, every isolation point AND THE STATE IT IS IN, every gas reading and when it was taken, every attachment and until when it is valid, and who signed for what. This is the only thing the model is trusted with, so every point lost here passes STRAIGHT THROUGH the recheck into a confirmed report. A declaration read wrong builds a correct checklist for the wrong hazards; an isolation read as isolated when it is open returns a confident SATISFIED.

MethodHow the test was run

python3 -m evals.run --run-id <id> [--floor rules]; each reading field is compared whole against data/gold.jsonl's reading block for that package.

Every grader on these pages scored the same 560 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, and what the grader made of it

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: data/gold.jsonl's reading block, computed by src/rules.py from the generator's injected facts and asserted by evals/check_labels.py to appear, as strings, in the package's own text.

No true/false rates for this grader. It records 7 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? Exact match on the whole field, no tolerance. A field is right or it is not.

Cadence: Every run, on every arm. It is the half no downstream code can repair.

The decisionWhen to reach for it