The business caseThe problem this solves

A bank's regulatory-change desk receives everything its supervisors publish, and most of it does not apply. Somebody reads each item, decides whether it reaches this charter, this asset size, these products and these lines of business, and writes down which clause made that decision. The volume is the whole problem: the reading is not hard, there is just a great deal of it, and the two ways it goes wrong are opposite. Read scope into silence and you open a change record nobody owes. Miss an exemption and you spend a quarter complying with something you were carved out of. the first pass a regulatory-change analyst makes over each published issuance -- reading the scope and exemption clauses against the institution's own profile and writing down, per dimension, whether it reaches us and which clause says so.

The inputThe actual issuance packets

The corpus is 50 issuance packets, 0.21 MB (txt 50). BECAUSE REAL REGULATORY TEXT CANNOT GO IN AN MIT REPO, AND SHOULD NOT. In most of the places official text is published it is copyright-encumbered or carries republication terms this licence cannot satisfy -- and worse, a file reproducing a real agency's real words under a real docket number, sitting in a folder of synthetic data, is a fabricated official record the moment somebody copies it out. So the five issuing bodies, the dockets, the section numbers and every sentence are invented, and every packet opens with a Synthetic Record banner. What is reproduced is the SHAPE -- header, plain-language summary, recitals, scope clause, exemptions, numbered operative paragraphs, dates -- because applicability turns on exactly those parts. Generating it also makes the label mechanical: the generator DEALS a (criterion_state, clause_kind) pair per test, WRITES the clause that makes it true, and then reads it back through the shipped rulebook to get the call. Gold is not somebody's reading of a clause; it is the clause, plus a lookup.

Synthetic Record
----------------
This is a SYNTHETIC regulatory-change packet generated by tools/build_corpus.py for the AI Foundry use-case kit `reg-applicability`. The issuing body, the docket reference, every section number, every sentence of the issuance and the institution it is routed to are INVENTED. Nothing here was copied from, adapted from or checked against any statute, regulation, register notice, supervisory letter, agency bulletin or bank procedure, because none was consulted. It describes no real institution, no real regulator and no real obligation, and it must never be treated as regulatory text.

Routing Slip
------------
Received by the regulatory change desk: 2026-01-03
Source feed: Federal Deposit Guarantee Board daily issuance feed
Routed to institution profile: PROF-A
Assessment owed: applicability, four tests, one worksheet

Internal Reference
------------------
Change ticket: RCM-2026-4100
Assigned analyst: N. Farhadi
Analyst direct extension: x2001
Analyst mailbox: farhadi@internal.example
Supervisory examiner of record: Examiner L. Constantine
Confidential supervisory correspondence reference: CSC-2026-2200
Committee routing: Regulatory Change Committee, standing item 4

Issuance Header
---------------
Issuing body: Federal Deposit Guarantee Board (FDGB)
Docket: FDGB-2026-0106
Instrument: Supervisory guidance
Published: 2026-01-03
Title: Prepaid access programme oversight

Summary
-------

The outcomeWhat a good result looks like

Four worksheet rows per issuance, each with the deciding clause reference the reader can open, and an issuance-level call an analyst can either accept or argue with. On this corpus that is 97.7 pct of 1000 cells right and 98.68 pct of the 151 clause citations landing on the exact deciding clause.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• First-pass applicability triage over a published issuance feed, for a regulatory-change analyst who will read every worksheet before acting on it — the one tier measured here -- 97.7 pct of 1000 cells, 97.5 pct of 200 test calls, and 48 of 48 exemption rows read in the right sense
  against the free keyword floor's 47.2 pct cells and 37.5 pct test calls, and against its 21 exemption rows read backwards. It also cites the exact deciding clause on 98.68 pct of the 151 rows that owe one, against the floor's 33.11 pct -- so a reviewer can check a row in one keystroke instead of re-reading the packet.
• Deciding whether the pure-code escalation flag can be relied on — it for RECALL and not for precision -- 7 of 7, with 3 false positives
  it missed nothing. Every packet where the issuance reaches the institution AND an exemption was not available to it was escalated, which is the direction that matters: a missed off-ramp is a quarter of compliance nobody owed.

And where nothing here is good enough:

• Running this on your own feed with no labelled set, and wanting a free quality signal — nothing on this page -- and that is a measured answer rather than a shrug
  the no-gold consistency diagnostic re-runs the rulebook over the model's OWN two stated facts and compares it with the call the model also stated. It caught 0 of the run's 5 call errors, and 0 of the free floor's 125. Every wrong row was internally consistent and wrong about the packet.
• Deciding whether a cheaper or a more deliberating tier would do better — nothing -- this page cannot answer it
  one tier was run, once. There is no second model and no repeat run, so every figure here is a single draw from one configuration.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-reg-applicability. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — A fresh clone with no key configured rebuilds the whole corpus and its gold in well under a second, passes the gold gate, scores the free keyword floor over all 50 packets, and serves the UI on 127.0.0.1:8974 with every recorded result browsable -- clicking Assess applicability returns a plain sentence saying nothing was called. Measured end to end at 0.34 seconds for build + gate + free floor on this machine. What it cannot do without a key is assess a NEW packet or reproduce run r001-reg-applicability.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. END-TO-END per packet inside evals/run.py: segmentation, selection, profile load, prompt assembly, ONE completion call, parse, clause location and the pure-code escalation rule. Everything but the call is under 3 ms, so these figures are the provider's latency plus rounding. Measured at 12 concurrent workers; the whole run was 260.6 seconds of wall clock. The longest single call was 167125 ms.

Current processWhat it replaces

the first pass a regulatory-change analyst makes over each published issuance -- reading the scope and exemption clauses against the institution's own profile and writing down, per dimension, whether it reaches us and which clause says so.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one packet section is

cut into named sections on underlined headings (src/segment.py), then the union of the sections each field maps to is sent (src/select.py). NOTHING IS CHUNKED AND NOTHING IS RETRIEVED -- the selected sections of one packet go whole into one call.

SetupWhat the setup figure measured

THERE IS NO INDEX. The figure is the cost of SEGMENTING the whole corpus -- 50 packets into 540 sections in 0.011 seconds of pure Python, measured on this machine. Nothing is embedded, nothing is stored and nothing is retrieved, so there is no build to amortise and a $0.00 here is real rather than unmeasured.

LicenceLicence

MIT -- this repository's own licence, corpus included

Bring your ownBring your own issuance packets

Two files and one dict. (1) Replace data/profiles.json with YOUR institution -- charter type, total consolidated assets, the closed list of products offered and lines of business conducted. (2) Drop your packets into data/corpus/ as .txt with underlined section headings, and point SECTION_HINTS in src/select.py at your section names -- and put whatever your desk stamps on the front, which must never leave the machine, into NEVER_SENT. (3) If your applicability tests are not charter / asset size / product / activity, edit data/rulebook.json and data/fields.json together. Then delete data/gold.jsonl and score by hand, or write your own labels in the same shape.

⚠︎ And what stops being true when you do: EVERY SCORE ON THIS PAGE STOPS BEING TRUE THE MOMENT YOU DO. Not because the model changes, but because the corpus is denser in decisive clauses than a real feed -- an average packet here addresses 3.0 of the 4 dimensions and a real issuance queue is mostly silence, so the 49 of 200 not_addressed rows measured here understate the share a real desk sees. And the one class this run cannot do is exactly the silent one. Re-run the eval on your own labels; nothing here transfers.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 4,876 tokens assembled, 1,625 are rulebooks — 33% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on ISS-0001 -- the same packet the raw reply above came from -- so the five parts sum to exactly the 4876 input tokens. Five calls at max_tokens=1, each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. kits/UC0070-reg-applicability/results/tokens-p002-reg-applicability.json. ⚑ THE FIXED HALF IS 4055 OF 4876 TOKENS (83.2 pct) and is paid on every one of the 50 calls.

Full promptThe prompt in full

You read a REGULATORY ISSUANCE PACKET -- one published issuance as a bank's regulatory change desk receives it -- together with the profile of ONE institution, and you return an APPLICABILITY WORKSHEET: does this issuance reach this institution, and which clause makes it so. You return JSON and nothing else.

You are preparing a worksheet for a regulatory-change analyst. You never open a change record, never assign an owner, never set a compliance date, never file anything, never waive anything and never clear an issuance as inapplicable. Your job is to say what the issuance's clauses state, apply the rulebook given below, and NAME THE DIMENSIONS THE ISSUANCE NEVER ADDRESSES.

RULES, in order of importance:
1. EXACTLY FOUR TEST ROWS, T1 T2 T3 T4, IN THAT ORDER, ON EVERY PACKET. T1 is CHARTER, T2 is ASSET SIZE, T3 is PRODUCT, T4 is ACTIVITY. A dimension the issuance never mentions is a row reading `not_addressed`, never an absent row: a worksheet that drops silent dimensions cannot be told apart from one that never checked them.
2. AN EXEMPTION INVERTS THE SENSE. A clause in an exemption or exclusion section, or phrased 'this part does not apply to', REMOVES an institution that meets its condition. So `criterion_met` on an exemption clause means the institution is OUT, and `criterion_not_met` on one means the institution stays IN. This is the branch a confident reader gets backwards.
3. FAILING TO QUALIFY FOR AN EXEMPTION IS NOT BEING OUT OF SCOPE. It is `exemption_not_available`, and it is the reason the issuance still bites. Never `out_of_scope`.
4. THE SUMMARY AND THE BACKGROUND DECIDE NOTHING. They name the subject in plain language and they name products and lines of business, and they are not clauses. Cite the clause that DECIDES -- in the scope section, the exemption section, or an operative paragraph of the substantive requirements. Never cite the Summary, the Background or the Distribution Notes.
5. `criterion_met` AND `criterion_not_met` ARE CLAIMS ABOUT THE INSTITUTION PROFILE BELOW, not about institutions in general. Check the clause's criterion against the profile's own charter type, its stated total consolidated assets, its list of products offered and its list of lines of business conducted.
6. `clause_kind` SAYS WHERE THE CRITERION IS STATED, and it is what makes rule 2 checkable. 'scope_clause' for the section that says who the issuance applies to; 'exemption_clause' for a section or clause that removes institutions; 'operative_paragraph' for a criterion stated inside a substantive requirement where no scope or exemption clause settles the dimension; 'none' only where the issuance says nothing on the dimension.
7. `test_call` IS DECIDED ONLY BY THE RULEBOOK BELOW, from `criterion_state` and `clause_kind`. Work through the six steps IN ORDER and STOP at the first that fires. In particular a criterion met inside an OPERATIVE PARAGRAPH is 'in_scope_by_operation' and never plain 'in_scope'.
8. `applicability` IS DECIDED ONLY BY THE RULEBOOK, from `instrument_type` and the four test calls. Decide it AFTER the four rows. An `interim_final_rule` is IN FORCE and is never 'monitor_only'.
9. 'not_addressed' IS A REAL ANSWER AND YOU ARE EXPECTED TO USE IT. Most issuances are silent on most dimensions. A dimension marked in scope on silence creates work nobody owes, and a dimension marked out of scope on silence closes an issuance that was never assessed.
10. Copy every reference verbatim from the packet -- the docket in `issuance_ref`, the clause reference in `clause_ref`. Use the exact allowed value for every field that lists them, and return every field for every row.

APPLICABILITY RULEBOOK (the authority for `test_call` and `applicability`; this is an ILLUSTRATIVE rulebook written for this kit, and it reproduces no statute, regulation, supervisory guidance or company procedure)

WHAT IS ON THE WORKSHEET
  - EXACTLY FOUR TEST ROWS, T1 through T4, on every issuance. A dimension the issuance never mentions is a row reading `not_addressed`, never an absent row - a worksheet that drops silent dimensions cannot be distinguished from one that never checked them.
  - ONE CLAUSE REFERENCE PER TEST, verbatim as the issuance prints it, and `none` when nothing addresses the dimension. A verdict with no clause behind it is an opinion.
  - THE CLAUSE THAT DECIDES, NOT THE CLAUSE THAT MENTIONS. A summary paragraph that names a product is not the clause that decides whether the product is covered; the scope or exemption clause is. Where both exist, cite the deciding one.
  - AN EXEMPTION IS READ IN ITS OWN SENSE. Meeting the condition of an exemption removes the institution. Failing to meet it leaves the institution in and is worth saying out loud.
  - THE INSTRUMENT TYPE IS READ OFF THE ISSUANCE, NOT GUESSED FROM ITS TONE. A proposed rule reads exactly like a final one for pages at a time; the header and the comment period are what settle it.

THE FOUR TESTS
  T1   CHARTER - the entity or charter type the issuance reaches
  T2   ASSET SIZE - an asset-size criterion the issuance sets
  T3   PRODUCT - a product or instrument the issuance covers
  T4   ACTIVITY - a line of business or activity the issuance covers

INSTRUMENT TYPES
  final_rule           a rule adopted in final form, binding when its compliance date arrives
  interim_final_rule   a rule adopted and already in force while comment is still being taken. IT IS BINDING - the word interim describes the procedure, not the obligation
  guidance             a supervisory statement of expectation. Not a rule, and still an expectation the institution is examined against
  proposed_rule        a rule offered for comment. It imposes nothing yet
  request_for_comment  a request for views, with no text proposed for adoption. It imposes nothing yet

THE TWO STATED FACTS, read off the issuance for every test
  criterion_state:
    criterion_met        the issuance states a criterion on this dimension and THIS institution meets it - the charter it names is this institution's charter, the asset threshold it sets is one this institution's assets satisfy, the product it covers is one this institution offers, the activity it covers is one this institution conducts
    criterion_not_met    the issuance states a criterion on this dimension and THIS institution does NOT meet it
    not_addressed        the issuance states no criterion on this dimension at all. It neither reaches this institution on it nor excludes it on it
  clause_kind:
    scope_clause         the criterion is stated in a clause that says who the issuance applies to
    exemption_clause     the criterion is stated in a clause that REMOVES something from the issuance - an exemption, an exclusion, a carve-out or a safe harbour. The sense is inverted: meeting the condition of an exemption puts an institution OUT
    operative_paragraph  the criterion is stated inside a substantive requirement rather than in a scope or exemption clause. The paragraph says who must do the thing, and there is no general scope clause that settles it
    none                 the issuance carries no clause on this dimension. This is the only value allowed when criterion_state is not_addressed, and the only criterion_state allowed with it

THE TEST CALL -- work through IN ORDER, stop at the first that fires
  1. NOTHING SAID. If clause_kind is `none` or criterion_state is `not_addressed`, the worksheet records `not_addressed`. An issuance that never mentions a dimension neither reaches the institution on it nor excludes it on it, and an analyst who reads silence as either is inventing text.
  2. THE EXEMPTION FIRES. If clause_kind is `exemption_clause` and criterion_state is `criterion_met`, the worksheet records `carved_out`. THE SENSE IS INVERTED HERE AND THIS IS THE BRANCH A CONFIDENT READER GETS BACKWARDS: an exemption whose condition the institution MEETS removes the institution, so a match on the words is the opposite of a match on the effect.
  3. THE EXEMPTION IS THERE AND DOES NOT REACH. If clause_kind is `exemption_clause` and criterion_state is `criterion_not_met`, the worksheet records `exemption_not_available`. There was an off-ramp and this institution does not qualify for it. That is NOT the same as being out of scope - it is the reason the issuance still bites - and it is the row counsel most often wants to argue.
  4. THE SCOPE DOES NOT REACH. If criterion_state is `criterion_not_met`, the worksheet records `out_of_scope`. The issuance states a criterion on this dimension and this institution does not meet it.
  5. REACHED THROUGH AN OPERATIVE PARAGRAPH. If clause_kind is `operative_paragraph`, the worksheet records `in_scope_by_operation`. The criterion is met, and it is met inside a substantive requirement rather than in a scope clause - so a reviewer has to confirm the paragraph is one that binds this institution rather than describing a class it happens to fall in. Never plain `in_scope`: collapsing the two hides the hop somebody has to re-walk.
  6. ANYTHING ELSE. The worksheet records `in_scope`.

THE APPLICABILITY CALL -- work through IN ORDER, stop at the first that fires
  1. If any test call is `carved_out`, the worksheet records `excluded_by_exemption`. An exemption that fires ends the assessment; nothing further is owed on this issuance.
  2. If any test call is `out_of_scope`, the worksheet records `does_not_apply`. A stated criterion this institution fails puts it out, whatever else the issuance reaches.
  3. If instrument_type is `proposed_rule` or `request_for_comment`, the worksheet records `monitor_only`. The issuance would reach this institution and it imposes nothing yet. `interim_final_rule` is NOT in this branch - it is in force.
  4. If no test call is `in_scope` or `in_scope_by_operation`, the worksheet records `indeterminate`. The issuance addresses none of the four dimensions in a way that reaches this institution, and it does not exclude it either. That is a finding about the text, not a gap in the assessment.
  5. If no test call is `in_scope` - every positive one is `in_scope_by_operation` - the worksheet records `applies_subject_to_review`. Nothing in a scope clause puts this institution in; an operative paragraph does, and somebody has to read it.
  6. ANYTHING ELSE. The worksheet records `applies`.

WHY `not_addressed` IS A REAL ANSWER
  An applicability worksheet that never says `not_addressed` is not a thorough assessment, it is a confident one. Most issuances are silent on most dimensions, and the value of the worksheet is that an analyst can see, in one column, which dimensions the text actually settles and which ones somebody read into it. Over-confidence costs in both directions at once: a dimension marked in scope on silence creates work nobody owes, and a dimension marked out of scope on silence closes an issuance that was never assessed.

INSTITUTION PROFILE (the institution this assessment is for; every `criterion_met` in your reply is a claim about a fact on this list)

  Profile id            PROF-A
  Institution           Meridian Commercial Bank
  Charter type          federally chartered commercial bank
  Deposit insurance     insured
  Holding company       held by a registered bank holding company
  Total consolidated assets   $18.4 billion
  Branches              212
  Closed-end residential mortgage originations, in EACH of the two preceding calendar years   6200

  PRODUCTS OFFERED
    - residential mortgage lending
    - residential mortgage servicing
    - home equity lines of credit
    - credit card issuing
    - consumer deposit accounts
    - commercial real estate lending
    - small business lending

  PRODUCTS NOT OFFERED
    - agricultural lending
    - trust and fiduciary services
    - remittance transfers
    - prepaid access products
    - digital asset custody
    - international correspondent banking

  LINES OF BUSINESS CONDUCTED
    - retail branch banking
    - mortgage origination
    - mortgage servicing
    - card issuing
    - commercial banking
    - treasury management

  LINES OF BUSINESS NOT CONDUCTED
    - wealth and trust
    - agricultural banking
    - cross-border payments
    - third-party sponsorship of fintech programs

  THE FOUR LISTS ABOVE ARE EXHAUSTIVE. Every product and every line of business this assessment recognises appears on exactly one of them, so a criterion naming any of them is always either met or not met -- never unknown.

Return these:
- issuance_ref (string) -- the docket or issuance reference printed in the Issuance Header section, verbatim (for example ODS-2026-0114)
- instrument_type (enum) one of: final_rule, interim_final_rule, guidance, proposed_rule, request_for_comment -- what kind of instrument this is, read off the Issuance Header line 'Instrument:' and confirmed by whether a Comment Period section is present. 'interim_final_rule' is ALREADY IN FORCE -- the word interim describes the procedure, not the obligation -- and is never 'proposed_rule'
- scope_clause_ref (string) -- the reference of the FIRST NUMBERED CLAUSE in the issuance's scope or applicability section, verbatim as the issuance prints it (for example 'Section 2(a)'). Not the section's unnumbered preamble, and not a clause from any other section. Return the string 'none' when the issuance carries no scope or applicability section at all -- some issuances scope each requirement inside its own operative paragraph instead, and saying so is a real answer
- applicability (enum) one of: applies, applies_subject_to_review, does_not_apply, excluded_by_exemption, monitor_only, indeterminate -- the issuance-level call, decided STRICTLY by the shipped rulebook from `instrument_type` and the four test calls and nothing else -- so decide it AFTER the four test rows. IN ORDER, stopping at the first that fires. (1) any test call 'carved_out' -> 'excluded_by_exemption'. (2) any test call 'out_of_scope' -> 'does_not_apply'. (3) instrument_type 'proposed_rule' or 'request_for_comment' -> 'monitor_only' (NEVER for 'interim_final_rule', which is already in force). (4) no test call 'in_scope' or 'in_scope_by_operation' -> 'indeterminate'. (5) no test call 'in_scope' -> 'applies_subject_to_review'. (6) anything else -> 'applies'
- tests (array of exactly four objects, T1 T2 T3 T4 in order) -- each carrying:
    - test_id (enum) one of: T1, T2, T3, T4 -- which applicability test this row is. THIS IS THE KEY -- return EXACTLY FOUR rows, one per test, in order: T1 CHARTER (the entity or charter type the issuance reaches), T2 ASSET SIZE (an asset-size criterion the issuance sets), T3 PRODUCT (a product or instrument the issuance covers), T4 ACTIVITY (a line of business or activity the issuance covers)
    - criterion_state (enum) one of: criterion_met, criterion_not_met, not_addressed -- whether the issuance states a criterion on THIS dimension and whether THE INSTITUTION IN THE PROFILE BLOCK meets it. 'criterion_met' when it states one and this institution meets it; 'criterion_not_met' when it states one and this institution does not; 'not_addressed' when the issuance states no criterion on this dimension at all. Report what the criterion says and whether the profile matches it -- this is NOT on its own the applicability call, because an exemption inverts the sense
    - clause_kind (enum) one of: scope_clause, exemption_clause, operative_paragraph, none -- WHERE the criterion is stated. 'scope_clause' when it is in a clause saying who the issuance applies to; 'exemption_clause' when it is in a clause that REMOVES something -- an exemption, exclusion, carve-out or safe harbour, usually phrased 'does not apply to' or 'is exempt from'; 'operative_paragraph' when it is stated inside a substantive requirement and no scope or exemption clause settles the dimension; 'none' when the issuance carries no clause on this dimension. 'none' is allowed only with criterion_state 'not_addressed', and 'not_addressed' is allowed only with 'none'
    - clause_ref (string) -- the reference of the SINGLE clause that decides this test, verbatim as the issuance prints it (for example 'Section 2(c)(1)' or 'Section 4.3'). Cite the clause that DECIDES, not a summary paragraph that merely mentions the subject. Return the string 'none' when clause_kind is 'none'
    - test_call (enum) one of: in_scope, in_scope_by_operation, out_of_scope, carved_out, exemption_not_available, not_addressed -- this test's worksheet call, decided STRICTLY by the shipped rulebook from `criterion_state` and `clause_kind` and nothing else. Work through the six steps IN ORDER, stopping at the first that fires. (1) clause_kind 'none' or criterion_state 'not_addressed' -> 'not_addressed'. (2) exemption_clause + criterion_met -> 'carved_out': AN EXEMPTION WHOSE CONDITION THE INSTITUTION MEETS REMOVES IT. (3) exemption_clause + criterion_not_met -> 'exemption_not_available'. (4) criterion_not_met -> 'out_of_scope'. (5) operative_paragraph -> 'in_scope_by_operation', never plain 'in_scope'. (6) anything else -> 'in_scope'

Return a JSON object with exactly these top-level keys: issuance_ref, instrument_type, scope_clause_ref, applicability, tests
`tests` is an array of EXACTLY FOUR objects, T1 T2 T3 T4 in that order. It is never shorter: a dimension the issuance does not address is a row reading `not_addressed`.

ISSUANCE PACKET
---------------
Synthetic Record
----------------
This is a SYNTHETIC regulatory-change packet generated by tools/build_corpus.py for the AI Foundry use-case kit `reg-applicability`. The issuing body, the docket reference, every section number, every sentence of the issuance and the institution it is routed to are INVENTED. Nothing here was copied from, adapted from or checked against any statute, regulation, register notice, supervisory letter, agency bulletin or bank procedure, because none was consulted. It describes no real institution, no real regulator and no real obligation, and it must never be treated as regulatory text.

Routing Slip
------------
Received by the regulatory change desk: 2026-01-03
Source feed: Federal Deposit Guarantee Board daily issuance feed
Routed to institution profile: PROF-A
Assessment owed: applicability, four tests, one worksheet

Issuance Header
---------------
Issuing body: Federal Deposit Guarantee Board (FDGB)
Docket: FDGB-2026-0106
Instrument: Supervisory guidance
Published: 2026-01-03
Title: Prepaid access programme oversight

Summary
-------
The subject of this instrument is oversight of third-party programme managers. The Federal Deposit Guarantee Board has observed inconsistent practice across supervised institutions and sets out what it expects a record to contain, who must hold it and how long it must be kept. Institutions engaged in prepaid access products, and those whose credit card issuing operations touch the same records, have asked for clarity on the point.
THIS SUMMARY IS DESCRIPTIVE AND SETTLES NOTHING. Scope is settled by the Scope and Applicability Section below.

Scope and Applicability
-----------------------
This part applies to an institution that meets EVERY criterion stated in this Section. Where this Section states no criterion on a matter, that matter does not bear on whether this part applies.
Article II, paragraph 1 The institution is a federally chartered commercial bank or a federally chartered savings association.
Article II, paragraph 2 The institution has total consolidated assets of $3 billion or more, measured as of the most recent report of condition.

Exemptions and Exclusions
-------------------------
This Section removes institutions from this part. Where ANY paragraph of this Section reaches an institution, that institution is outside this part, whatever the scope clauses would otherwise require.
Article III, paragraph 1 This part does not apply to an institution that conducts agricultural banking.

Substantive Requirements
------------------------
Article V, paragraph 3 A record required by this part shall be retained in a form that permits retrieval within five business days of a supervisory request.
Article V, paragraph 4 An institution shall review the procedures adopted under this part at least annually and record the date of the review.
Article V, paragraph 5 An institution shall make a record required by this part available in a readable electronic form on request.
Article V, paragraph 6 An institution shall be able to identify the individual accountable for each record required by this part.
Article V, paragraph 7 Where a third party creates a record on an institution's behalf, the institution remains responsible for its contents and its availability.

Effective and Compliance Dates
------------------------------
Effective date: 2026-04-03.
Compliance date: 2026-04-03. An institution may adopt the practice earlier.

Distribution Notes
------------------
Circulated to the regulatory change desk and to the owners of the affected control library. Queries to the desk, not to the issuing body. Trade associations have signalled that they read this instrument as reaching every supervised institution; that reading is theirs and is not part of the issuance.

Raw responseThe raw response

{
  "issuance_ref": "FDGB-2026-0106",
  "instrument_type": "guidance",
  "scope_clause_ref": "Article II, paragraph 1",
  "applicability": "applies",
  "tests": [
    {
      "test_id": "T1",
      "criterion_state": "criterion_met",
      "clause_kind": "scope_clause",
      "clause_ref": "Article II, paragraph 1",
      "test_call": "in_scope"
    },
    {
      "test_id": "T2",
      "criterion_state": "criterion_met",
      "clause_kind": "scope_clause",
      "clause_ref": "Article II, paragraph 2",
      "test_call": "in_scope"
    },
    {
      "test_id": "T3",
      "criterion_state": "not_addressed",
      "clause_kind": "none",
      "clause_ref": "none",
      "test_call": "not_addressed"
    },
    {
      "test_id": "T4",
      "criterion_state": "criterion_not_met",
      "clause_kind": "exemption_clause",
      "clause_ref": "Article III, paragraph 1",
      "test_call": "exemption_not_available"
    }
  ]
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Decide whether a new rule reaches your bank — 50 issuance packets. One model answered, and every answer was then graded Six different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

There is no LLM judge in this kit and that is not a shortcut. Every cell is an exact string match after light normalisation, a set comparison over the four test ids, the shipped rulebook's own lookup re-run over the same facts, or a literal substring lookup of the cited clause inside the packet. A judge would add cost, variance and a second thing to validate, for a question the generator wrote one right answer to.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate -- a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid -- the real spend for this kit's build is recorded in the shared call ledger, not on this page.

Grading unitWhat the grading figure prices

Free means free: evals/judge.py makes no model call, and neither does the keyword floor it is compared against. Grading run r001-reg-applicability again costs $0.00 and takes under a second.

The gradersSix ways to grade

The free floor is not a null grader and not a straw man: it is the scope-section keyword scan a change desk already runs, it is perfect on three of the eight cell types, and it costs nothing. Everything the money buys on this page is the difference between matching words and reading a clause in its own sense.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same packet, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering costs money here and grading costs nothing

Every scorer in this kit is pure code: evals/judge.py, evals/check_labels.py, evals/baseline.py and tools/capture_example.py make no model call between them, so re-grading run r001-reg-applicability costs $0.00 and takes under a second. The bill is the pipeline. One full eval pass -- 50 packets, one call each -- is $0.756 on the projected card, and a forker re-running the eval pays that again and nothing on top. The kit's whole build cost 68 calls on the shared ledger: 50 scored, 6 calibration at max_tokens=24000 ($0.0921), 10 prompt-split probes at max_tokens=1 across two packets, and 2 screenshot calls.

Cost driversWhat actually moves the bill

• ⚑ PROVIDER-SIDE REASONING, WHICH IS THE BILL. 197,242 of the run's 211,708 output tokens (93.2 pct) were reasoning rather than the JSON worksheet, on a reply that is a few hundred tokens of content. Output is 84 pct of the cost on the projected card. Reasoning was left at the provider's default and nothing here has measured what turning it off would cost or break.
• NO RETRIEVAL AT ALL, which is why there is no top-k knob on the scenarios page and nothing to turn: the selected sections of one packet (456 of the corpus's 540) go whole into one call, and the input is fixed by the packet rather than by how much context anybody chose to pull.
• THE FIXED HALF, 4055 of 4876 input tokens (83.2 pct) on the example call, byte-identical on every one of the 50 calls: the system instruction (943), the applicability rulebook (1625), the institution profile (337) and the field schema (1150). The issuance itself is 821 tokens. Both of the two largest fixed parts are exactly what a forker replaces with something longer.
• OUTPUT LENGTH DOES NOT SCALE WITH PACKET LENGTH, WHICH IS THE FINDING THAT SURPRISED THIS BUILD. The calibration on the SIX LONGEST packets topped out at 5,967 output tokens; the scored run produced a reply of 21,215 -- 3.56x that -- on a corpus whose longest packet is 1.4x its shortest. The reasoning budget tracks how tangled the applicability question is, not how long the text is, and there is no cheap proxy to sort a corpus by that.

Your volumeWhat it costs at your volume

Linear in packets. Each call is independent, carries the same fixed prefix and shares nothing, so 500 packets is 10x $0.756 = $7.56 on the projected card, and 5,000 is 100x. Nothing amortises: there is no index to build, no embedding to reuse and no cache, so the 83.2 pct fixed prefix is paid every single time. The only sub-linear move available is prompt caching on the fixed half, and it has not been tried.

Model choiceWhy this model

It is the tier the shared key points at, and the run is one call per packet with no retrieval and no second pass, so the model choice IS the cost. Nothing here argues it is the right one -- see Eval.fitment: no second tier was run.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 issuance packets, one completion call each, one tier. The calibration, the two prompt-split measurements and the two screenshot calls are recorded separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

11 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pySegment

cut one packet into named sections on its underlined headings; falls back to a single whole-document segment when a packet has none. Pure code.

# Cut a trace file into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pySelect

map each field to the sections that can answer it, and send their union. Internal Reference is mapped by nothing and never leaves the machine. Pure code.

# Pick which sections of an issuance packet are sent. Pure code -- the last deterministic step

BANNER = "Synthetic Record"
ROUTING = "Routing Slip"
HEADER = "Issuance Header"
SUMMARY = "Summary"
SCOPE = "Scope and Applicability"
EXEMPT = "Exemptions and Exclusions"
SUBSTANTIVE = "Substantive Requirements"
DATES = "Effective and Compliance Dates"
COMMENT = "Comment Period"

src/profile.pyProfile

load the ONE institution the packet was routed to and render it into the prompt. The fixed half of the question. Pure code.

# THE INSTITUTION PROFILE -- the fixed half of this kit's question, loaded from data/profiles.json.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
PROFILES_PATH = os.path.join(HERE, "data", "profiles.json")
def load(path=PROFILES_PATH):
P = load()
BY_ID = {p["id"]: p for p in P["profiles"]}
PROFILE_IDS = tuple(p["id"] for p in P["profiles"])
PRODUCTS = tuple(P["products"])
ACTIVITIES = tuple(P["activities"])
CHARTERS = dict(P["charter_types"])

src/rulebook.pyRulebook

the applicability rule as DATA -- one definition read by the generator, the prompt and the scorer.

# THE APPLICABILITY RULEBOOK, loaded from data/rulebook.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
CRITERION_STATES = ("criterion_met", "criterion_not_met", "not_addressed")
CLAUSE_KINDS = ("scope_clause", "exemption_clause", "operative_paragraph", "none")
TEST_CALLS = ("in_scope", "in_scope_by_operation", "out_of_scope", "carved_out",
APPLICABILITY = ("applies", "applies_subject_to_review", "does_not_apply",
TEST_IDS = ("T1", "T2", "T3", "T4")
INSTRUMENT_TYPES = ("final_rule", "interim_final_rule", "guidance", "proposed_rule",
NOT_YET_BINDING = ("proposed_rule", "request_for_comment")

src/prompt.pyPrompt

assemble system + rulebook + institution profile + field schema + packet sections. One prompt per packet.

# Assemble the applicability prompt. ONE prompt per issuance packet, four test rows in the reply.

SYSTEM = (
VERDICTS = ("not_addressed", "in_scope", "in_scope_by_operation", "out_of_scope", "carved_out",
VERDICT_MEANINGS = {
RULEBOOK_TEXT = rulebook_block()
def field_schema(issuance_fields, test_fields):
def profile_id_of(secs):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/adapters/__init__.pyModel — a swap seam

ONE completion call per packet, max_tokens=24000. The whole AI layer.

You change it to: .env picks provider, base URL, key and model. Adding a vendor is one function and one entry in PROVIDERS; it must return token counts, because lens 05 publishes them and lens 07 prices them.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/assess.pyEscalation flag — a swap seam

needs_counsel_review, run afterwards in pure code over whatever the model returned -- never over gold.

You change it to: compute() is a POLICY, deliberately a different function from the rulebook lookup. Changing what gets escalated must be re-scored; it never changes what the rulebook says.

# Assemble one packet's applicability worksheet: segment, select, prompt, one model call, then a

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 24000
def load_fields():
def test_fields(fields=None):
def issuance_fields(fields=None):
def load_doc(doc_id):
def documents():

evals/check_labels.pyGold gate

re-derives every label from the shipped rulebook, floors every published class, re-measures the guardrail's firing band and audits selection on all 50 packets -- before any run may spend.

# THE GOLD GATE. Everything that must be true of the corpus, the labels and the selector BEFORE a

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
FLOORS = {"carved_out": 20, "exemption_not_available": 20, "in_scope_by_operation": 22,
APPL_FLOOR = 2
GUARD_BAND = (10.0, 60.0)
def load_gold():
def main():

evals/judge.pyScorers

five of them: row coverage, per-cell exact match, the two rulebook calls with the determinacy split, whether the cited clause EXISTS in the packet, and the escalation flag's confusion matrix.

# Score a run against gold. PURE CODE -- no model, no network, no judgement, $0.00.

ISSUANCE_CELLS = ("issuance_ref", "instrument_type", "scope_clause_ref", "applicability")
TEST_CELLS = ("criterion_state", "clause_kind", "clause_ref", "test_call")
def norm(v):
def _rows_by_id(rec):
def _val(row, name):
def score_coverage(records, golds):
def score(fields, records, golds):
def score_calls(records, golds):
CLAUSE_TOKEN = re.compile(

evals/baseline.pyFree floor

the scope-section keyword scan a desk already runs. No key, no tokens, $0.00.

# THE FREE FLOOR: the scope-section keyword scan a regulatory-change desk already runs.

MONEY = re.compile(r"\$(\d+(?:\.\d+)?)\s*(billion|million)", re.I)
CLAUSE = re.compile(
DOCKET = re.compile(r"^Docket:\s*(\S+)", re.M)
INSTRUMENT = re.compile(r"^Instrument:\s*(.+)$", re.M)
INSTRUMENT_MAP = [
def _threshold(text):
def _cell(v):
def assess(doc_text, fields):

src/app.pyLocal UI

one packet, one worksheet, on 127.0.0.1 only. Renders with no key.

# The minimal local UI. Standard library only -- python -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "8974"))
class H(BaseHTTPRequestHandler):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 4835 input and 4234 output tokens per packet, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's packets are entirely synthetic (tools/build_corpus.py, seed 20260822): no real regulator, no real issuance, no real docket, no real institution and no real person exists in the corpus, and nothing was fetched from anywhere. No statute, regulation, supervisory guidance or agency interpretation is reproduced or named anywhere in the kit, and every packet opens with a Synthetic Record banner so a single file copied out of the repository still says what it is. The only outbound traffic the kit makes is one chat-completion request per packet, carrying the mapped sections plus the fixed instruction, the shipped rulebook and the institution profile -- and the Internal Reference section, which is the one part of a real change packet a desk would most mind sending anywhere, is mapped by no field and never leaves the machine. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env, this kit's own .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser.

The experimentWe did not attack it -- and one boundary was broken until the gate caught it on 16 packets

Three boundaries hold. Two were confirmed by reading the code: no code path opens a change record, sets a date, files anything or writes anything, and the rulebook lookup and the escalation rule read only enums, so no packet text can reach either. The third is confirmed by a CHECK RATHER THAN BY READING, and it needed to be -- evals/check_labels.py asserts, before any run may spend, that the Internal Reference section is never sent and that the Summary always is, on all fifty packets and per field. With the fallback this kit inherited it convicts 16 of the 50 by name; with _fallback() it holds at 0 of 50, and both directions were run. The fourth is open. An indirect prompt injection needs a section an outside party authored, and a routing slip is exactly that. Confirmed by reading the code and by a pre-flight assertion that convicted it on 16 packets in fifty, not by an attack run, on 2026-08-22 -- no attack run was fired; see redteam.why.

Three boundaries hold and the second is the interesting one, because it did not hold when it was written: a fallback path every sibling kit ships sent the desk's own stamp -- analyst, extension, examiner, confidential correspondence reference -- on 16 packets in fifty, and it was caught by an assertion that runs before every scored run rather than by reading the code. The fourth is open and is named as open: a regulatory-change packet is a document several parties touch, which makes an instruction hidden in a routing slip or a distribution note exactly the attack this kit's shape invites, and nothing here has tried it.

The result0 attack trials, and three of four boundaries checked here hold on evidence -- one of them only after a pre-flight assertion convicted the code on 16 packets in fifty. The one that does not hold is the one this kit's shape invites: a regulatory-change packet is a document several parties touch, and whether an instruction hidden in a routing slip could move a call -- or, more cheaply, make a dimension read not_addressed -- is unmeasured.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted misdirection measures whether the worksheet reads the CLAUSE rather than the summary, whether it reads an exemption in its own sense, and whether it will say a dimension is not addressed. None of those measures whether it obeys an instruction hidden in the same packet, and the read-scope-into-silence attack is the one this kit's own failure mode suggests is cheapest.

HonestyWhat this does not prove

• Whether an indirect prompt injection in a routing slip or a distribution note could move a call or silence a dimension. No attack run exists, and the silence version is the one this run's own failure mode suggests would be easiest.
• Whether the kit behaves safely against a hostile provider -- a response body crafted to break the JSON extraction in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed document, and nothing beyond that was tested.
• Whether a real regulatory-change packet would carry anything sensitive this kit mishandles beyond the Internal Reference section. The corpus has no personal data by construction; a real packet carries named analysts, examiner correspondence, internal legal advice and the institution's own confidential supervisory material, and none of that path is exercised.
• Whether the institution profile is safe to send. It carries no customer data and no names of people by design, but total consolidated assets, charter type and a closed product list are institution-level facts a real bank may still not want in a third-party prompt, and nothing here measures what a redacted profile would do to the answers.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_counsel_review fires when the packet's composed applicability is `applies` or `applies_subject_to_review` AND at least one test row is `exemption_not_available` -- the issuance reaches this institution, and there was an off-ramp it does not qualify for. Both facts come out of the same reply; the rule is run afterwards in pure code, over whatever the model returned, and never over gold. A reply carrying no row the rule can read returns None rather than False -- an unknown is not a pass.

src/assess.py::compute(), called by assess() on every packet, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by identical code. The rulebook composition it reads lives in a SEPARATE module, src/rulebook.py, which is also what the corpus generator wrote gold with and what src/prompt.py states to the model -- one definition, three readers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a compliance decision, a change record, a filing, a waiver or a clearance. Nothing here decides anything about an issuance. Every call is a proposal with the rulebook's own reasoning attached, and a regulatory-change team and its counsel decide.
• It is NOT a check on whether the worksheet is RIGHT, and this run is the demonstration. The rule reads the reply; if the reply invented scope out of silence, the rule fires confidently on an issuance that settles nothing. That is 3 of the 10 packets it escalated.
• It is NOT any bank's regulatory-change procedure. A missed off-ramp is this kit's own simplification, invented for this corpus. No statute, regulation, supervisory guidance, bar association note or company procedure was consulted, and none is reproduced. A real desk weighs the size of the obligation, the cost of the argument, the institution's supervisory relationship and what counsel already advised on the same exemption last year.
• It is NOT a judgement about the rulebook. The flag reads the calls the rulebook produced; if the rulebook is wrong, the flag is confidently wrong with it. The shipped rulebook is illustrative and nobody qualified has reviewed it.

WatchedWhat is watched, and why that one

The kit measures 22 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

5 measured by the latest run 17 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 50, extraction_cells 1000, failures 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

2 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 2 runs measured the same system — they differ on provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 2 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/assess.py, on any change to data/rulebook.json (which moves the calls the flag reads), on any change to data/fields.json's allowed values for criterion_state, clause_kind or test_call, on any change to data/profiles.json (which moves what criterion_met means), and on any corpus regeneration -- the gate re-measures the firing band every time. Changing WHAT GETS ESCALATED is a policy change and must be re-scored, even though it never changes what the rulebook says.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the rulebook and the institution profile are JSON files read with json, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

One path per packet -- segment, select, load the profile, prompt, one call, parse, locate the clause, compose -- with no branch, no loop and no agent. The only shape worth noting is that the reply's LENGTH is fixed at four rows, which is why row coverage is a set comparison over a constant denominator rather than a recall figure over a list of unknown length.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain -- the JSON extraction from a fenced reply, the section regex, the exact-name matcher, the clause locator, the backoff policy and the .env reader are all code somebody has to own. This kit paid an instalment during its own build: the first draft of the clause-reference scheme let Section 2(a) nest inside Section 2(a)(2), so an exact-match citation grader would have been grading an ambiguous string. The generator's own assertion caught it on the first run.
• No framework means no framework's ecosystem -- no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. Nothing validates the reply's shape except src/prompt.py::parse, which fails closed to {} rather than to a worksheet with no rows, because on this kit an empty tests list is never a legitimate answer.
• Nothing checks that the model's clause reference is one the SECTION it names could contain. The citation grader checks the reference exists in the packet, and evals/judge.py's cited_a_real_clause_wrongly bucket is what catches the rest -- but that is a scoring observation after the fact, not a guard in the pipeline.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-reg-applicability on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 1 earlier run on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 1 run on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT -- this repository's own licence, corpus included Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one line⚑ Row coverage -- did all four dimensions come back

Every packet owes EXACTLY FOUR rows, T1 through T4, and a dimension the issuance never mentions is a row reading not_addressed rather than an absent row. This grader is a set comparison over (packet, test id) with the two directions named and never averaged: a MISSED row is a dimension the worksheet is silent about, and a PHANTOM row is a test id the reply invented. On this shape a short reply is the cheapest way to look tidy while saying nothing, and a worksheet that drops silent dimensions cannot be told apart from one that never checked them.

MethodHow the test was run

evals/judge.py::score_coverage, in-process, no key and no model. The same function scores the free floor, because a floor and a real run scored by different scorers cannot be compared honestly.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

reconstructed from results/eval-r001-reg-applicability.json — the run's non-hit cells overwritten onto gold. No model call was made to produce this file.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the four test ids in data/rulebook.json's rules, asserted present and in order on every gold row by evals/check_labels.py before any run may spend. The denominator is fixed at four per packet by construction, so a low recall means exactly one thing.

How tight can the band be? No band and no tolerance -- a set comparison over a fixed four-element key.

Cadence: Re-score on any change to data/rulebook.json, data/fields.json, src/prompt.py, data/profiles.json or tools/build_corpus.py -- the first two move the answer key, the third moves what the model was told and the last two move the corpus. Re-run evals/run.py (paid, 50 calls) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one linePer-cell exact match, light normalisation

Does the model's value match gold on each of the four issuance-level cells and each of the four cells on each of the four test rows -- 20 cells a packet, 1,000 over the corpus -- after trimming whitespace, collapsing spaces, lowercasing and dropping trailing punctuation? THE DENOMINATOR IS GOLD'S ROWS, NOT THE RUN'S: a row the run never returned scores as a miss on all four of its cells rather than disappearing.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

reconstructed from results/eval-r001-reg-applicability.json — the run's non-hit cells overwritten onto gold. No model call was made to produce this file.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, which is the (criterion_state, clause_kind) pair the generator DEALT and then wrote the clause for, plus the clause reference it printed. evals/check_labels.py asserts every field is populated and in its allowed set on all 200 rows, that clause_kind and criterion_state agree about silence, and that every clause reference is printed verbatim and exactly once in the packet it labels -- before any run is allowed to spend.

How tight can the band be? No tolerance band -- exact match after trimming and lowercasing, never a continuous score to round.

Cadence: Re-score on any change to data/rulebook.json, data/fields.json, src/prompt.py, data/profiles.json or tools/build_corpus.py -- the first two move the answer key, the third moves what the model was told and the last two move the corpus. Re-run evals/run.py (paid, 50 calls) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe two rulebook calls, six-way and six-way

Did the run land on the same calls the shipped rulebook produces -- in_scope / in_scope_by_operation / out_of_scope / carved_out / exemption_not_available / not_addressed per test row, and applies / applies_subject_to_review / does_not_apply / excluded_by_exemption / monitor_only / indeterminate per packet? The test call is SCORED ON MATCHED ROWS ONLY, and saying so is half of what makes the number honest: a call about a row the run never returned has already been counted, once, as a coverage failure. The issuance call is scored on every packet, because it is owed once per packet whatever the rows did.

MethodHow the test was run

evals/judge.py::score_calls, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

reconstructed from results/eval-r001-reg-applicability.json — the run's non-hit cells overwritten onto gold. No model call was made to produce this file.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's calls are re-derived inside the grader by the same lookup the kit publishes -- src/rulebook.py, reading data/rulebook.json -- so the truth this grades against can never be a separately-typed label that drifted from the rule.

How tight can the band be? No band -- a six-way categorical against a lookup, with the full confusion matrix recorded in the run file rather than an accuracy alone.

Cadence: Re-score on any change to data/rulebook.json, data/fields.json, src/prompt.py, data/profiles.json or tools/build_corpus.py -- the first two move the answer key, the third moves what the model was told and the last two move the corpus. Re-run evals/run.py (paid, 50 calls) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one line⚑ Four directions: silence, over-caution, the operative hop and the inverted exemption

Four errors that a single accuracy figure averages away, each on its own denominator and none of them netted against another. OVER-CONFIDENT: of every dimension the issuance is SILENT on, how many did the run answer anyway -- scope read into silence, which creates work nobody owes. OVER-CAUTIOUS: its mirror, a dimension the issuance does address called not_addressed, which closes an issuance nobody assessed. THE OPERATIVE HOP: an in_scope_by_operation row collapsed into plain in_scope, which is over-confidence not about WHETHER the issuance reaches you but about WHERE it says so. THE INVERTED EXEMPTION: on a row whose deciding clause REMOVES institutions, did the run get the direction right -- because on an exemption a match on the words is the opposite of a match on the effect, and this is the one call the whole kit exists to measure.

MethodHow the test was run

evals/judge.py::score_calls, the determinacy block, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

reconstructed from results/eval-r001-reg-applicability.json — the run's non-hit cells overwritten onto gold. No model call was made to produce this file.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The same rulebook lookup as the grader above, with the rows partitioned four ways by what GOLD says -- silent / addressed / operative / exemption. All four directions come out of one pass over the same 200 rows.

How tight can the band be? Four rates, four denominators, printed side by side and never averaged. The smallest denominator here is 27 rows, which makes that rate wide, and the page says so rather than rounding it.

Cadence: Re-score on any change to data/rulebook.json, data/fields.json, src/prompt.py, data/profiles.json or tools/build_corpus.py -- the first two move the answer key, the third moves what the model was told and the last two move the corpus. Re-run evals/run.py (paid, 50 calls) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one line⚑ Does the cited clause exist in the packet, and is it the deciding one

THREE VERDICTS, AND THE MIDDLE ONE IS THE POINT. A reference that names the exact deciding clause is checkable in one keystroke. A reference that names a real clause of the real packet and the WRONG one is a row somebody will open, read and disagree with -- expensive but self-correcting. A reference that names nothing in the packet at all is a row nobody can check, and it looks exactly like the good one on the page. A single accuracy figure folds the second and third together, which is the difference between an argument and a fabrication.

MethodHow the test was run

evals/judge.py::score_citation, in-process, no key and no model. It re-reads the packet off disk rather than trusting anything in the reply.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

reconstructed from results/eval-r001-reg-applicability.json — the run's non-hit cells overwritten onto gold. No model call was made to produce this file.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Two things at once: gold's clause reference for the exact verdict, and the PACKET TEXT ITSELF for the locates verdict -- a literal substring lookup, no normalisation. tools/build_corpus.py asserts every gold reference is printed verbatim and exactly once, and that no reference printed in a packet nests inside another, so a substring hit cannot be a coincidence.

How tight can the band be? No band. Exact is normalised (whitespace, case, trailing punctuation); locates is a raw substring lookup against the packet as shipped.

Cadence: Re-score on any change to data/rulebook.json, data/fields.json, src/prompt.py, data/profiles.json or tools/build_corpus.py -- the first two move the answer key, the third moves what the model was told and the last two move the corpus. Re-run evals/run.py (paid, 50 calls) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineneeds_counsel_review against the same rule run over gold

Does the pure-code escalation decision -- the issuance reaches this institution AND the packet carries an exemption it does not qualify for -- land on the same packets it would land on if every row had been read perfectly? It is a BUSINESS CONDITION, so unlike the self-consistency diagnostic it genuinely needs labels, and saying so is half of what makes the number believable.

MethodHow the test was run

evals/judge.py::score_flag, in-process, no key and no model. The same function also computes the no-gold consistency diagnostic beside it.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

reconstructed from results/eval-r001-reg-applicability.json — the run's non-hit cells overwritten onto gold. No model call was made to produce this file.

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/assess.py::compute(), the same function the run uses, applied to GOLD's own rows. One rule, one input shape, so a change to the rule moves both sides together and the grader cannot silently grade an old policy.

How tight can the band be? No threshold -- a boolean condition over three enums, with the full 2x2 recorded in the run file. None (the reply carried no readable row) is a third state and is counted apart from false; it was 0 on this run.

Cadence: Re-check on any change to compute() in src/assess.py, on any change to data/rulebook.json (which moves the calls the flag reads), on any change to data/fields.json's allowed values, and on any corpus regeneration -- the gate re-measures the firing band every time. Changing WHAT GETS ESCALATED is a policy change and must be re-scored even though it never changes what the rulebook says.

The decisionWhen to reach for it