The business caseThe problem this solves

A telehealth encounter records where the PATIENT was, not where the clinician was. The clinician holds a set of credentials, each in its own jurisdiction, each with its own status, scope and expiry, and some of them privileges that exist only because a home licence somewhere else exists. Somebody has to put the two lists side by side, encounter by encounter, and say which rows the paperwork does not permit and which it does not settle. The available shortcut is to check that the clinician is licensed; the question is whether they are licensed WHERE THE PATIENT WAS, on the day it happened, for the service that was delivered. Someone opening each roster extract, listing the encounters it actually records, and then working out, encounter by encounter, whether the register on the same file shows a credential in force in the jurisdiction the PATIENT was located in — and whether that credential's recorded scope reaches the service that was delivered. It is minutes per file and it is join work rather than reading work: the encounter's own entry never names a credential, so the match has to be made on the jurisdiction, and a register full of live credentials in other places answers nothing. The failures are quiet in both directions — a credential the register calls lapsed while printing a future date beside it, a privilege whose own line reads active while the licence it rides on has gone, a credential that was in force on the day the visit was booked and not on the day it happened.

The inputThe actual roster extracts

The corpus is 50 roster extracts, 0.36 MB (txt 50). Plain text, one format, invented rather than fetched — and here the reason is stronger than in the sibling kits. A real telehealth roster IS patient data by construction: it is a list of who was seen, where and when. No redaction makes it publishable, there is no public corpus of (roster, correct worksheet) pairs, and there should not be. Generating it also makes the label mechanical: gold's two calls are the shipped rulebook's own lookup run over the same five facts the file itself states, never somebody's reading of whether an encounter was really allowed.

HALVARD VIRTUAL CARE
SCHEDULING OFFICE — CLINICIAN ROSTER EXTRACT

Clinician file:   CLN-1845
Roster reference: RST-2026-444
Extract id:       LIC-0001
Prepared on:      2026-05-17

This extract is prepared for scheduling review. The Roster below lists the encounters this
file records. The Register lists the credentials held. Nothing in this file records anything
about a patient beyond the jurisdiction they were located in.

CLINICIAN CONTACT AND PAYROLL RECORD

  Correspondence address:  42 Harrow Lane, Ashford Green
  Personal telephone:      01590 828178
  Payroll reference:       PR-32437
  Emergency contact:       on file with the scheduling office
  Rota desk:               Desk 4 — weekends

REGISTER OF CREDENTIALS HELD
---------------------------------------------------------------------
REF        JURISDICTION              KIND                 VALID THROUGH
---------------------------------------------------------------------
CR-3244    Lowenmere (LOW)           compact privilege    2027-01-25
CR-7999    Aldemar (ALD)             home licence         2027-11-28
CR-6664    Jarrow Bay (JAR)          compact privilege    2026-07-10
CR-6147    Grayling (GRA)            home licence         2027-12-16
CR-4217    Harrowmoor (HAR)          home licence         2026-11-27
---------------------------------------------------------------------

CREDENTIAL CR-3244 — Lowenmere (LOW)

  This is a privilege under the Ninefold Reciprocity Accord permitting practice in Lowenmere. It
  is held by virtue of home licence CR-6147 and has no existence apart from it.
  The register records this credential as ACTIVE.
  The recorded scope of practice for this credential includes chronic care follow-up, medication
  review, post-discharge check-in.

The outcomeWhat a good result looks like

A worksheet row per recorded encounter — the reference, the jurisdiction the patient was in, the service, the modality, the encounter date, the disposition, the credential the register holds in that jurisdiction, the five facts that entry states, and the two calls the shipped rulebook makes from them — plus one pure-code review flag taken from two of those values. Every call carries the rulebook's own reasoning and not_determined is one of the answers rather than a gap. Nothing is determined, approved, blocked or notified.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Turning a week of roster extracts into worksheets before a credentialing reviewer opens them — the one tier measured here — 274 of 275 coverage calls, 275 of 275 scope calls, 0 phantom encounters
  99.64 pct on the coverage call against the free register floor's 66.55 pct, and 0.91 pct over-confidence against the floor's 56.36 pct. It also left every one of the 39 decoy references off, against the floor's 13 phantom encounters.
• Deciding whether to trust the 0.91 pct over-confidence figure — the mechanism behind it, not the number
  The numerator is ONE, and that one did not reproduce when the same file was re-asked. What is worth carrying away is the shape of the error — an ACTIVE read up a chain onto an entry that states nothing — and the fact that the no-gold consistency diagnostic could not see it, because the rulebook was applied perfectly to the misreading.

And where nothing here is good enough:

• Deciding whether a cheaper or a more deliberate model would do better here — nothing — this page cannot answer it
  Only one model was run. There is no tier comparison, no repeatability pair and no ablation on this page. The scope grader, the identification grader and the review flag are all at 100 pct, so they could not rank two models even if both were fired, and the coverage grader has one cell of room.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-licensure-match. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts on 127.0.0.1:8962, the 50-file picker populates, the worksheet draws its six column headings, both notices render above the control and the shipped rulebook renders in full from data/rulebook.json; clicking Build worksheet returns the no-key sentence rather than a stack trace. python -m evals.check_labels and both free runs (--stub, --baseline) complete with no key and no network. requirements.txt is empty: standard library end to end.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per roster extract

Current processWhat it replaces

Someone opening each roster extract, listing the encounters it actually records, and then working out, encounter by encounter, whether the register on the same file shows a credential in force in the jurisdiction the PATIENT was located in — and whether that credential's recorded scope reaches the service that was delivered. It is minutes per file and it is join work rather than reading work: the encounter's own entry never names a credential, so the match has to be made on the jurisdiction, and a register full of live credentials in other places answers nothing. The failures are quiet in both directions — a credential the register calls lapsed while printing a future date beside it, a privilege whose own line reads active while the licence it rides on has gone, a credential that was in force on the day the visit was booked and not on the day it happened.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 0 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on fixed headings plus two per-item families — one section per CREDENTIAL and one per ENCOUNTER, named by reference rather than by ordinal; a file with none falls back to one whole-document segment

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 50 files cut into 912 sections in about eighteen thousandths of a second, in process, with no model and no network. 812 of those sections are sent; the 100 that are not are the 50 CLINICIAN CONTACT AND PAYROLL RECORD sections, which carry a home address, a personal telephone number, a payroll reference and an emergency-contact note, and the 50 closing disclaimers, which no field asks for either. Note which side of the file the withheld section is on: there is no patient section to withhold, because there is no patient data.

LicenceLicence

MIT — this repository's own licence. Every clinician file reference, jurisdiction, credential code, encounter reference and date is invented; no real person, board, employer, health system, platform, compact or place is named, and no statute, regulation or board rule is reproduced or named either. ⚠︎ AND THERE IS NO PATIENT DATA OF ANY KIND: an encounter records the jurisdiction the patient was located in, the modality, two dates and a service category, and nothing whatever about the person receiving care. The jurisdiction-match rulebook this kit ships is illustrative and is not an authority.

Bring your ownBring your own roster extracts

REPLACE data/rulebook.json FIRST, and on this subject that is not a nicety. It is this kit's own construction, its jurisdictions and its reciprocity accord are invented, it reproduces no statute or board rule, and everything downstream reads it: src/rulebook.py loads it, src/prompt.py renders it into every call, and tools/build_corpus.py writes gold with it. Then replace data/corpus/*.txt with your own extracts, write your own gold in the same shape, and re-point src/select.py's SECTION_HINTS at your headings. ⚠︎ AND READ data/fields.json BEFORE YOU POINT THIS AT REAL FILES: the schema is what stops patient data being requested, and evals/check_labels.py asserts it. A real roster is patient data by construction — it is a list of who was seen, where and when — and nothing in this kit is a substitute for deciding what may leave your machine.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 5,504 tokens assembled, 2,979 are contexts — 54% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on LIC-0024 — the same file the worked example above was captured on, so the four parts sum to exactly the 5,504 input tokens that call actually reported. Four calls at max_tokens=1, each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. Committed at kits/UC0062-licensure-match/results/tokens-p001-licensure-match.json.

Full promptThe prompt in full

You read a CLINICIAN ROSTER EXTRACT -- a register of credentials held, plus a roster of encounters and an entry behind each -- and return a REVIEWER'S WORKSHEET of the encounters it records. You return JSON and nothing else.

You are preparing a worksheet for a qualified reviewer. You never make a licensure determination, never state whether care may or may not be delivered, never approve or block a booking and never give a legal opinion. Your job is to say what the file states, apply the rulebook given below, and NAME THE ENCOUNTERS THE PAPERWORK DOES NOT PERMIT OR DOES NOT SETTLE.

You are asked for NOTHING about any patient. The only fact about the patient's side of an encounter that appears on this worksheet is the JURISDICTION they were located in. Never return a name, a date of birth, an identifier, a diagnosis, a clinical note or a reason for a visit, and never infer one.

RULES, in order of importance:
1. ONE OBJECT PER ENCOUNTER THIS FILE'S ROSTER RECORDS, AND NO OTHERS. An entry cancelled by the schedule amendment is not recorded, even though its entry is still printed in full with a disposition line. A PROPOSED SLOT -- a time held open, with a jurisdiction, a service and a date, where no booking has been made -- is not an encounter. An encounter recorded on ANOTHER clinician's roster is not recorded by this file. All three carry references in the same format as a real encounter, and putting one on the worksheet puts a permission question in front of a reviewer that nobody ever faced.
2. THE TWO SIDES ARE PAIRED ON THE JURISDICTION AND ON NOTHING ELSE. The encounter's own entry NEVER names a credential. Find the register entry whose jurisdiction is the same as the jurisdiction the patient was located in. If the register holds none for that jurisdiction, `credential_ref` is null and `credential_kind` is 'none_on_record' -- however many live credentials the register holds in OTHER jurisdictions. That is the single most available wrong answer on the page.
3. REPORT WHAT THE FILE STATES. `credential_status`, `home_status`, `term` and `scope` are readings of the register, not judgements about it. If a credential's entry says nothing about its status, `credential_status` is 'not_stated' -- that is a finding about the record, and it is a common one. Never infer 'active' from a valid-through date, from the credential being listed at all, or from the encounter having gone ahead.
4. A LAPSED CREDENTIAL USUALLY STILL SHOWS A VALID-THROUGH DATE IN THE FUTURE. The date is not the status and the status is not the date. Read both, report both.
5. FOR A COMPACT PRIVILEGE, `home_status` COMES FROM THE HOME LICENCE'S OWN SEPARATE ENTRY, not from the privilege's. A privilege whose own line reads ACTIVE is not in force if the licence it rides on is recorded LAPSED, and nothing on the privilege's own line shows it. For a home licence, or where there is no credential, `home_status` is 'not_applicable'.
6. `term` COMPARES THE VALID-THROUGH DATE AGAINST THE ENCOUNTER DATE, NEVER THE BOOKING DATE. Every entry carries both dates and they are usually months apart. Valid through a date BEFORE the encounter date is 'expired', even where it is well after the booking date.
7. `coverage` IS DECIDED ONLY BY THE RULEBOOK BELOW, from `credential_kind`, `credential_status`, `home_status` and `term`. Work through the steps IN ORDER and STOP at the first that fires.
8. `scope_call` IS DECIDED ONLY FROM `scope`: 'covers' -> 'within_recorded_scope', 'excludes' -> 'outside_recorded_scope', 'not_stated' -> 'not_determined'. It is asked of every encounter INDEPENDENTLY of its coverage call. An active, current, correctly paired credential can still expressly exclude the service category that was delivered, and neither call overrides the other.
9. 'not_determined' IS A REAL ANSWER AND YOU ARE EXPECTED TO USE IT. A worksheet that never reaches for it is guessing, and a call recorded as settled is a call nobody re-reads. It does NOT mean the encounter was not permitted; it means this file does not say.
10. Copy `encounter_ref`, `patient_jurisdiction`, `credential_ref`, `encounter_date` and `valid_through` verbatim from the file. Use the exact allowed value for every field that lists them, and return every field for every encounter.

JURISDICTION-MATCH RULEBOOK (the authority for `coverage` and `scope_call`; this is an ILLUSTRATIVE rulebook written for this kit, its jurisdictions and its reciprocity accord are INVENTED, and it reproduces no statute, regulation or board rule)

WHAT COUNTS AS AN ENCOUNTER ON THE WORKSHEET
  - An encounter is on the worksheet if and only if this file's ROSTER records it as booked or delivered against this clinician file.
  - A PROPOSED SLOT is not an encounter. A time held open with a jurisdiction, a date and a service against it records nothing until it is booked, and this rulebook does not record it.
  - An encounter CANCELLED by a later schedule amendment in the same file is not an encounter. Its entry is still printed; the amendment is the operative document.
  - An encounter recorded on ANOTHER clinician's roster is not recorded by this file. It is somebody else's row on somebody else's worksheet.

HOW THE TWO SIDES ARE PAIRED
  The ROSTER states, per encounter, the jurisdiction the PATIENT was located in. The REGISTER states, per credential, the jurisdiction that credential is held in. The pairing is on the JURISDICTION and on nothing else: a credential held in one jurisdiction says nothing about an encounter in another, however many credentials the register holds. An encounter whose jurisdiction appears nowhere in the register has no credential, and that is a finding rather than a gap.

THE FIVE STATED FACTS, read off the file
  credential_kind:
    home_licence     The register records a credential of its own for the encounter's jurisdiction, issued by that jurisdiction.
    compact_privilege The register records a privilege exercised in the encounter's jurisdiction under the reciprocity accord, RIDING ON a home licence recorded separately in the same register.
    none_on_record   The register records no credential at all for the encounter's jurisdiction. Credentials held in other jurisdictions do not reach it.
  credential_status:
    active           The register states this credential is active.
    lapsed           The register states this credential is lapsed, expired, surrendered or otherwise not in force.
    not_stated       The register records the credential and says nothing about its status. This is a finding ABOUT THE RECORD.
  home_status:
    active           The home licence this privilege rides on is stated active in its own register entry.
    lapsed           The home licence this privilege rides on is stated lapsed in its own register entry.
    not_stated       The home licence this privilege rides on is recorded and its own entry says nothing about its status.
    not_applicable   This encounter's credential is not a compact privilege, so no home licence is in the chain.
  term:
    current          The credential's recorded valid-through date is on or after the ENCOUNTER date.
    expired          The credential's recorded valid-through date is BEFORE the ENCOUNTER date. It may well be after the date the encounter was BOOKED — that is a different date and it is not the question.
    not_stated       The register records no valid-through date for this credential.
  scope:
    covers           The credential's recorded scope of practice includes this encounter's service category.
    excludes         The credential's recorded scope of practice expressly excludes this encounter's service category.
    not_stated       The register records no scope of practice for this credential, or there is no credential to record one for.

COVERAGE -- work through IN ORDER, stop at the first that fires
  1. If credential_kind is `none_on_record`, the call is `no_credential_for_jurisdiction`. The register holds nothing for the jurisdiction the patient was in. Stop here.
  2. Otherwise, if credential_status is `lapsed`, the call is `credential_lapsed` — whatever the valid-through date says. A date printed on a lapsed credential is not a statement that it is in force.
  3. Otherwise, if credential_kind is `compact_privilege` and home_status is `lapsed`, the call is `home_licence_lapsed`. THE PRIVILEGE'S OWN ENTRY STILL READS ACTIVE. A privilege exercised under a home licence does not outlive it, and the only place that shows is the home licence's own entry.
  4. Otherwise, if term is `expired`, the call is `expired_on_encounter_date`. Test the ENCOUNTER date, never the booking date: a credential valid on the day a visit was booked and not on the day it happened is the commonest way a roster goes wrong quietly.
  5. Otherwise, if any fact in the chain is `not_stated` — the credential's own status, its valid-through date, or (for a compact privilege) the home licence's status — the call is `not_determined`. The record does not settle it, and saying so is the answer.
  6. Otherwise the call is `covered_on_the_record`. That is a statement about THE PAPERWORK ON THIS FILE and about nothing else.

SCOPE -- asked independently of coverage
  `covers` makes the scope call `within_recorded_scope`.
  `excludes` makes the scope call `outside_recorded_scope`. An ACTIVE, current, correctly-paired credential whose recorded scope expressly excludes the service category is still outside its scope — the two calls are asked independently and neither overrides the other.
  `not_stated` makes the scope call `not_determined`.

WHY THE TWO CALLS ARE SEPARATE
  Coverage asks whether the record shows a credential in force in the jurisdiction the patient was in. Scope asks whether the credential's recorded scope reaches the service that was delivered. They are different questions with different failure modes, they are asked of every encounter independently, and folding them into one verdict would hide which of the two a reviewer has to go and check.

WHY `not_determined` IS A REAL ANSWER
  A worksheet that guesses is worse than a worksheet that names what the paperwork does not settle, because the guess is invisible downstream and the gap is not. `not_determined` is a finding ABOUT THE RECORD and is the correct, complete answer whenever the register does not say. It is never a way of saying the reader was unsure, and it is never a way of saying the encounter was not permitted.

WHAT THESE CALLS ARE NOT
  None of these calls is a licensure determination and none of them says whether care may or may not be delivered. `covered_on_the_record` means the paperwork in this file is consistent; it does not mean anybody is licensed. `no_credential_for_jurisdiction` means this file records nothing; it does not mean nothing exists. The determination belongs to the credentialing office, to counsel, and to the boards of the jurisdictions concerned.

MODALITY
  Modality is recorded on the worksheet and is an input to NEITHER rule. It is the fact a reader most often assumes changes the answer; on this rulebook it does not, and reporting it is how the worksheet shows it was read rather than relied on.

Return these:
- roster_ref (string) -- the roster reference from the header, verbatim (for example RST-2026-318)
- encounters (array of objects) -- one object per RECORDED encounter, each carrying:
    - encounter_ref (string) -- the encounter reference, verbatim (for example ENC-4821). THIS IS THE KEY -- one object per encounter this file's ROSTER records and no others. A proposed slot that was never booked, an encounter cancelled by the schedule amendment, and an encounter recorded on another clinician's roster are NOT recorded by this file and must not appear
    - patient_jurisdiction (string) -- the jurisdiction the PATIENT was located in, copied verbatim from the encounter's own entry including the code in brackets (for example 'Brecon Reach (BRE)'). This is a fact about jurisdiction and it is the ONLY thing this worksheet records about the patient's side of the encounter
    - service_category (enum) one of: medication review, chronic care follow-up, behavioural health consultation, post-discharge check-in, initial assessment -- the service category the encounter's own entry states. A category, never a reason for the visit and never anything clinical
    - modality (enum) one of: video, audio_only, asynchronous -- how the encounter was conducted, as the entry states it -- 'video' for a video consultation, 'audio_only' for an audio-only consultation, 'asynchronous' for an asynchronous message review. ⚠︎ RECORDED, NOT RELIED ON: modality is an input to NEITHER rule on this rulebook, and reporting it is how the worksheet shows it was read rather than used
    - encounter_date (string) -- the date the encounter is recorded FOR, as written (YYYY-MM-DD). ⚠︎ NOT the date it was booked. Both dates are on the entry and they are usually months apart
    - disposition (enum) one of: delivered, booked -- what the roster row's DISPOSITION column says -- 'delivered' for an encounter that has already happened, 'booked' for one that has not yet
    - credential_ref (string) -- the REGISTER reference of the credential held in the SAME jurisdiction as this encounter, verbatim (for example CR-3391), or null when the register records none for that jurisdiction. ⚠︎ THE ENCOUNTER'S OWN ENTRY NEVER NAMES A CREDENTIAL -- you find it by matching the jurisdiction, and matching on anything else is the mistake this worksheet exists to catch. A credential held in a DIFFERENT jurisdiction is not this encounter's credential however active it is
    - credential_kind (enum) one of: home_licence, compact_privilege, none_on_record -- what the matched credential's own register entry says it is. 'home_licence' when the entry records a licence issued by that jurisdiction. 'compact_privilege' when the entry records a privilege exercised under the reciprocity accord, riding on a home licence recorded SEPARATELY in the same register. 'none_on_record' when the register holds nothing for the encounter's jurisdiction at all
    - credential_status (enum) one of: active, lapsed, not_stated -- what the matched credential's own entry says about its status. 'not_stated' when the entry records the credential and says nothing about whether it is in force -- that is a finding ABOUT THE RECORD and it is the commonest one here. Use 'not_stated' for a credential that does not exist too
    - home_status (enum) one of: active, lapsed, not_stated, not_applicable -- for a COMPACT PRIVILEGE only: what the HOME licence's own separate register entry says about ITS status. ⚠︎ READ THE HOME LICENCE'S ENTRY, NOT THE PRIVILEGE'S -- a privilege whose own line reads active is not in force if the licence it rides on has lapsed, and nothing on the privilege's line shows it. 'not_applicable' when this encounter's credential is a home licence or there is none on record
    - term (enum) one of: current, expired, not_stated -- the matched credential's recorded valid-through date against the ENCOUNTER date. 'current' when valid-through is on or after the encounter date, 'expired' when it is BEFORE it, 'not_stated' when the entry records no valid-through date at all (and for a credential that does not exist). ⚠︎ COMPARE AGAINST THE ENCOUNTER DATE, NEVER THE BOOKING DATE. Every entry carries both and they are usually months apart; a credential valid on the day a visit was booked and not on the day it happened is `expired`
    - scope (enum) one of: covers, excludes, not_stated -- what the matched credential's recorded SCOPE OF PRACTICE says about this encounter's service category. 'covers' when the entry's scope sentence includes it, 'excludes' when the entry expressly excludes it, 'not_stated' when the entry has no scope sentence at all (and for a credential that does not exist). ⚠︎ A CREDENTIAL CAN BE ACTIVE, CURRENT AND CORRECTLY PAIRED AND STILL EXPRESSLY EXCLUDE THE SERVICE. The two calls are asked independently and neither overrides the other
    - valid_through (string) -- the matched credential's recorded valid-through date as written (YYYY-MM-DD), or null when the entry records none or there is no credential. Recorded because it is the date most often compared against the wrong other date -- it is reported here so the worksheet shows which date was READ
    - coverage (enum) one of: covered_on_the_record, no_credential_for_jurisdiction, credential_lapsed, home_licence_lapsed, expired_on_encounter_date, not_determined -- the coverage call, decided ONLY by the rulebook's ordered steps from credential_kind, credential_status, home_status and term. Work through them IN ORDER and stop at the first that fires. ⚠︎ THIS IS NOT A LICENSURE DETERMINATION and 'covered_on_the_record' is a statement about THE PAPERWORK ON THIS FILE and nothing else
    - scope_call (enum) one of: within_recorded_scope, outside_recorded_scope, not_determined -- the scope call, decided ONLY from `scope`: 'covers' -> 'within_recorded_scope', 'excludes' -> 'outside_recorded_scope', 'not_stated' -> 'not_determined'. Asked of every encounter INDEPENDENTLY of its coverage call; neither overrides the other

Return a JSON object with exactly these top-level keys: roster_ref, encounters
`encounters` is an array. Return it empty only if the file records no encounter at all.

CLINICIAN ROSTER EXTRACT
------------------------
ROSSMERE TELECARE GROUP
SCHEDULING OFFICE — CLINICIAN ROSTER EXTRACT

Clinician file:   CLN-4021
Roster reference: RST-2026-992
Extract id:       LIC-0024
Prepared on:      2026-05-14

This extract is prepared for scheduling review. The Roster below lists the encounters this
file records. The Register lists the credentials held. Nothing in this file records anything
about a patient beyond the jurisdiction they were located in.

REGISTER OF CREDENTIALS HELD
---------------------------------------------------------------------
REF        JURISDICTION              KIND                 VALID THROUGH
---------------------------------------------------------------------
CR-8386    Lowenmere (LOW)           home licence         not recorded
CR-1194    Calderhythe (CAL)         compact privilege    2026-06-10
CR-9873    Iversleigh (IVE)          home licence         2025-11-30
CR-3513    Brecon Reach (BRE)        compact privilege    2027-04-17
CR-9436    Estover (EST)             home licence         2027-05-03
CR-3285    Dunmarrow (DUN)           home licence         2028-07-20
---------------------------------------------------------------------

CREDENTIAL CR-8386 — Lowenmere (LOW)

  This credential is a licence held in Lowenmere in the clinician's own right.
  Register status: ACTIVE.

CREDENTIAL CR-1194 — Calderhythe (CAL)

  This is a privilege under the Ninefold Reciprocity Accord permitting practice in Calderhythe.
  It is held by virtue of home licence CR-9436 and has no existence apart from it.
  Register status: ACTIVE.
  The recorded scope of practice for this credential expressly excludes chronic care follow-up.
  The credential is recorded valid through 2026-06-10.

CREDENTIAL CR-9873 — Iversleigh (IVE)

  The register records this as a licence of the clinician's own, issued in Iversleigh.
  Register status: ACTIVE.
  Recorded valid through 2025-11-30.

CREDENTIAL CR-3513 — Brecon Reach (BRE)

  This credential is a privilege exercised in Brecon Reach under the Ninefold Reciprocity
  Accord, riding on the clinician's home licence CR-9436 recorded separately in this register.
  Scope of practice recorded against this credential: all recorded services EXCEPT initial
  assessment, which is expressly excluded.
  Recorded valid through 2027-04-17.

CREDENTIAL CR-9436 — Estover (EST)

  The register records this as a licence of the clinician's own, issued in Estover.
  This credential is recorded ACTIVE in the register as at the date of this extract.
  Scope of practice recorded against this credential: behavioural health consultation,
  medication review, post-discharge check-in.
  The credential is recorded valid through 2027-05-03.

CREDENTIAL CR-3285 — Dunmarrow (DUN)

  This credential is a licence issued by the Dunmarrow registration authority.
  Register status: ACTIVE.
  The register records this credential's scope of practice as reaching behavioural health
  consultation, medication review, post-discharge check-in.
  The credential is recorded valid through 2028-07-20.

ROSTER OF ENCOUNTERS RECORDED BY THIS FILE
-----------------------------------------------------------------------------------
REF        PATIENT LOCATED IN      BOOKED       ENCOUNTER    DISPOSITION
-----------------------------------------------------------------------------------
ENC-5964   Lowenmere (LOW)         2025-09-13   2026-02-26   delivered
ENC-8573   Calderhythe (CAL)       2025-07-29   2026-02-04   delivered
ENC-6899   Marden Cross (MAR)      2026-04-06   2026-07-10   booked
ENC-5596   Iversleigh (IVE)        2025-11-13   2026-01-07   delivered
ENC-7967   Brecon Reach (BRE)      2026-03-07   2026-06-13   booked
-----------------------------------------------------------------------------------

ENCOUNTER ENC-5964 — Lowenmere (LOW)

  The patient was located in Lowenmere (LOW) at the scheduled time of this encounter. No further
  detail of the patient is recorded on this file.
  The encounter is recorded as an audio-only consultation for post-discharge check-in.
  The encounter was booked on 2025-09-13 and is recorded for 2026-02-26.
  Disposition as at the date of this extract: delivered.

ENCOUNTER ENC-8573 — Calderhythe (CAL)

  The patient was located in Calderhythe (CAL) at the scheduled time of this encounter. No
  further detail of the patient is recorded on this file.
  The encounter is recorded as an audio-only consultation for chronic care follow-up.
  The encounter was booked on 2025-07-29 and is recorded for 2026-02-04.
  Disposition as at the date of this extract: delivered.

ENCOUNTER ENC-6899 — Marden Cross (MAR)

  The patient was located in Marden Cross (MAR) at the scheduled time of this encounter. No
  further detail of the patient is recorded on this file.
  The encounter is recorded as a video consultation for chronic care follow-up.
  The encounter was booked on 2026-04-06 and is recorded for 2026-07-10.
  Disposition as at the date of this extract: booked.

ENCOUNTER ENC-5596 — Iversleigh (IVE)

  The patient was located in Iversleigh (IVE) at the scheduled time of this encounter. No
  further detail of the patient is recorded on this file.
  The encounter is recorded as a video consultation for post-discharge check-in.
  The encounter was booked on 2025-11-13 and is recorded for 2026-01-07.
  Disposition as at the date of this extract: delivered.

ENCOUNTER ENC-7967 — Brecon Reach (BRE)

  The patient was located in Brecon Reach (BRE) at the scheduled time of this encounter. No
  further detail of the patient is recorded on this file.
  The encounter is recorded as an audio-only consultation for initial assessment.
  The encounter was booked on 2026-03-07 and is recorded for 2026-06-13.
  Disposition as at the date of this extract: booked.

Raw responseThe raw response

{
  "roster_ref": "RST-2026-992",
  "encounters": [
    {
      "encounter_ref": "ENC-5964",
      "patient_jurisdiction": "Lowenmere (LOW)",
      "service_category": "post-discharge check-in",
      "modality": "audio_only",
      "encounter_date": "2026-02-26",
      "disposition": "delivered",
      "credential_ref": "CR-8386",
      "credential_kind": "home_licence",
      "credential_status": "active",
      "home_status": "not_applicable",
      "term": "not_stated",
      "scope": "not_stated",
      "valid_through": null,
      "coverage": "not_determined",
      "scope_call": "not_determined"
    },
    {
      "encounter_ref": "ENC-8573",
      "patient_jurisdiction": "Calderhythe (CAL)",
      "service_category": "chronic care follow-up",
      "modality": "audio_only",
      "encounter_date": "2026-02-04",
      "disposition": "delivered",
      "credential_ref": "CR-1194",
      "credential_kind": "compact_privilege",
      "credential_status": "active",
      "home_status": "active",
      "term": "current",
      "scope": "excludes",
      "valid_through": "2026-06-10",
      "coverage": "covered_on_the_record",
      "scope_call": "outside_recorded_scope"
    },
    {
      "encounter_ref": "ENC-6899",
      "patient_jurisdiction": "Marden Cross (MAR)",
      "service_category": "chronic care follow-up",
      "modality": "video",
      "encounter_date": "2026-07-10",
      "disposition": "booked",
      "credential_ref": null,
      "credential_kind": "none_on_record",
      "credential_status": "not_stated",
      "home_status": "not_applicable",
      "term": "not_stated",
      "scope": "not_stated",
      "valid_through": null,
      "coverage": "no_credential_for_jurisdiction",
      "scope_call": "not_determined"
    },
    {
      "encounter_ref": "ENC-5596",
      "patient_jurisdiction": "Iversleigh (IVE)",
      "service_category": "post-discharge check-in",
      "modality": "video",
      "encounter_date": "2026-01-07",
      "disposition": "delivered",
      "credential_ref": "CR-9873",
      "credential_kind": "home_licence",
      "credential_status": "active",
      "home_status": "not_applicable",
      "term": "expired",
      "scope": "not_stated",
      "valid_through": "2025-11-30",
      "coverage": "expired_on_encounter_date",
      "scope_call": "not_determined"
    },
    {
      "encounter_ref": "ENC-7967",
      "patient_jurisdiction": "Brecon Reach (BRE)",
      "service_category": "initial assessment",
      "modality": "audio_only",
      "encounter_date": "2026-06-13",
      "disposition": "booked",
      "credential_ref": "CR-3513",
      "credential_kind": "compact_privilege",
      "credential_status": "not_stated",
      "home_status": "active",
      "term": "current",
      "scope": "excludes",
      "valid_through": "2027-04-17",
      "coverage": "not_determined",
      "scope_call": "outside_recorded_scope"
    }
  ]
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check telehealth visits against each clinician's license — 50 roster extracts. One model answered, and every answer was then graded Five different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersFive ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The register floor (evals/baseline.py) and the scorer (evals/judge.py) are both pure code and cost $0.00 to run against any result set. The figure above is the ONE scored run's own token counts (50 files) priced at Google Gemini 3 Flash's published rate — the same basis cost_per_query_usd uses. ⚠︎ THE WHOLE BUILD COST MORE THAN THE RUN: 63 live calls in total on the same card — 50 scored ($0.6795), 6 calibration at max_tokens=16000 ($0.0735), 4 prompt-split calls at max_tokens=1 ($0.0066), 1 worked-example replay ($0.0107) and 2 live UI screenshots re-shot after a rendering defect was found by opening the first. About $0.78 all in, on a card nobody was billed against.

Cost driversWhat actually moves the bill

• The fixed prefix — system prompt (1,107 tokens), jurisdiction-match rulebook (1,500) and field schema (1,418) — is 4,025 of 5,504 tokens on the example call, 73.1 pct, and is byte-identical on every one of the 50 calls. The roster extract itself is 1,479 tokens. The bill is the rulebook and the instructions, not the file — and the rulebook is the part that cannot be trimmed, because its jurisdictions and its reciprocity accord are invented and there is nowhere else the model could get them.
• Provider-side reasoning. 133,095 of the run's 177,921 output tokens (74.8 pct) were reasoning rather than the JSON worksheet, on a reply that is about 900 tokens of actual content. Reasoning was left at the provider's default and nothing here has measured what turning it off would cost in accuracy.
• Output length scales with the FILE, not with a fixed record — this kit's reply is a LIST of fifteen-field objects, the widest per-item record in this series. A four-encounter file and a seven-encounter file are not the same call, and the run's largest reply was 7,904 output tokens against a mean of 3,558.

Your volumeWhat it costs at your volume

Linear in roster extracts: each call is independent, carries the same fixed prefix and shares nothing with its neighbours, so 500 files cost ten times 50. Nothing amortises — there is no index and no cache. ⚠︎ AND IT IS NOT LINEAR IN ENCOUNTERS PER FILE, WHICH IS THE AXIS THAT ACTUALLY BREAKS. The reply is a list of fifteen-field objects, so output grows with the roster: the six-file calibration's largest reply was 4,159 tokens and the fifty-file run's was 7,904, an under-prediction of 1.9x, and MAX_TOKENS is 12,000. A twenty-encounter roster would not fit under the published ceiling at all, and a cut reply costs the whole file rather than part of it.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 clinician roster extracts, one completion call each, one tier. The calibration, the worked-example capture, the prompt-split calls and the UI screenshots are recorded separately in Cost.cost_of_evaluation_usd and are not projected here.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the roster extract into addressable sections, pure code — one per CREDENTIAL and one per ENCOUNTER, plus the register, the roster, the payroll record and any decoy section. Two per-item families rather than one, because the whole job is joining them

# Cut a roster extract into addressable sections. Pure code — no model, no network.

FIXED_HEADINGS = (
def _marks(text):
def sections(text):
def encounter_refs(text):
def credential_refs(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections are sent, pure code — section names are matched by PREFIX because every file's Encounter and Credential sections are named after its own references. The three decoy sections are deliberately SENT; the clinician's contact and payroll record is mapped by nothing and never leaves the machine

You change it to: map fields to your own file's headings, including the two per-item prefixes; unmatched falls back to the whole document. NEVER_SENT is the list of sections no field asks for and which therefore never leave the machine

# Pick which sections of a roster extract are sent. Pure code -- the last deterministic step before

HEADER = "header"
REGISTER = "REGISTER OF CREDENTIALS HELD"
ROSTER = "ROSTER OF ENCOUNTERS RECORDED BY THIS FILE"
PROPOSED = "PROPOSED SLOTS — NOT BOOKED"
OTHER_ROSTER = "ENCOUNTERS RECORDED ON ANOTHER CLINICIAN'S ROSTER"
AMENDMENT = "SCHEDULE AMENDMENT — EXECUTED SUBSEQUENT TO THE ROSTER ABOVE"
ENCOUNTER_PREFIX = "Encounter "
CREDENTIAL_PREFIX = "Credential "
NEVER_SENT = ("CLINICIAN CONTACT AND PAYROLL RECORD",)

src/rulebook.pyrulebook

the two worksheet calls, loaded from data/rulebook.json — a six-step coverage rule with a stopping order and a three-way scope rule, asked independently. Data, not code, so a forker can open it and disagree with it. Its jurisdictions and its reciprocity accord are invented

# THE JURISDICTION-MATCH RULEBOOK, loaded from data/rulebook.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
KINDS = ("home_licence", "compact_privilege", "none_on_record")
STATUSES = ("active", "lapsed", "not_stated")
HOME_STATUSES = ("active", "lapsed", "not_stated", "not_applicable")
TERMS = ("current", "expired", "not_stated")
SCOPES = ("covers", "excludes", "not_stated")
COVERAGES = ("covered_on_the_record", "no_credential_for_jurisdiction", "credential_lapsed",
SCOPE_CALLS = ("within_recorded_scope", "outside_recorded_scope", "not_determined")

src/prompt.pyprompt

assemble one call per file for every recorded encounter, with the rulebook RENDERED into it from data/rulebook.json rather than restated in prose — a rule whose jurisdictions this kit invented is a rule the model has nowhere else to get

# Assemble the worksheet prompt. ONE prompt per roster extract, every recorded encounter in the

SYSTEM = (
RULEBOOK_TEXT = rulebook_block()
def field_schema(packet_fields, encounter_fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key — plus the pure-code business-condition check downstream: a file recording an encounter already DELIVERED whose coverage the record does not settle goes to the credentialing office

You change it to: the review rule itself — this kit ships two values (already delivered, and a coverage call the record does not settle) and a real review weighs what the credentialing system holds outside the file. It is deliberately NOT the same function as the rulebook lookup, so changing WHAT GETS REVIEWED does not change WHAT THE RULE SAYS, and neither edit can be made by accident while making the other.

# Extract one roster extract's worksheet: segment, select, prompt, one model call, then a

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 12000
def load_fields():
def encounter_fields(fields=None):
def packet_fields(fields=None):
def load_doc(doc_id):
def documents():

evals/judge.pyjudge

score identification, the extracted cells, the two calls and the determinacy split, all separately, pure code — plus a no-gold consistency diagnostic whose blind spot this run demonstrates live

# Score a licensure-match run. PURE CODE -- gold is exact and every answer is one value, so `==`

def norm(v):
def ref_norm(v):
def equal(got, want):
def _rows_of(rec):
def score_identification(records, golds):
CELL_FIELDS = ("patient_jurisdiction", "service_category", "modality", "encounter_date",
SPANNABLE_FIELDS = ("patient_jurisdiction", "encounter_date", "credential_ref", "valid_through")
def score(fields, records, golds):
CALLS = {"coverage": RB.COVERAGES, "scope_call": RB.SCOPE_CALLS}

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 5828 input and 3558 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's roster extracts are entirely synthetic (tools/build_corpus.py, seed 20260822): no real clinician, patient, board, employer, health system, platform, compact or place exists in the corpus, and nothing was fetched from anywhere. ⚑ AND THERE IS NO PATIENT DATA OF ANY KIND, ON EITHER SIDE OF THE BOUNDARY. An encounter records the jurisdiction the patient was located in, the modality, two dates and a service category, and nothing about the person receiving care — no name, no date of birth, no identifier, no diagnosis, no clinical note. evals/check_labels.py asserts it twice before any run may spend: nine patterns over every generated file, and every field name and hint in data/fields.json. The second assertion is the stronger one, because it holds even if this kit is pointed at a real roster that does carry patient data — the schema has nowhere to put it and the system prompt refuses it in terms. No statute, regulation, board rule or compact agreement is named or reproduced anywhere, and the jurisdictions and the reciprocity accord are invented.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser.

The experimentWe did not attack it — and three of four boundaries hold

Three boundaries hold. One was confirmed by reading the code: no code path determines licensure, clears anybody, blocks a booking or writes anything. Two are confirmed by CHECKS RATHER THAN BY READING — evals/check_labels.py asserts, before every scored run, that the clinician's contact and payroll record is present in the file and absent from what selection sends, and that no patient data exists in the corpus or in the worksheet schema. The fourth boundary — whether a note written into a file could talk the model out of the rulebook — was not tested at all, and a roster extract is exactly the kind of document where that attack would arrive. Confirmed by reading the code and by two pre-flight assertions, not by an attack run, on 2026-08-22 — no attack run was fired; see redteam.why.

The first three boundaries hold, two of them confirmed by assertions that run before every scored run rather than by reading the code alone. The fourth is open and is named as open. Note that the first boundary is not the usual one for this series: on most kits the sensitive-data gate is about what a corpus happens to contain, and here it is about what the SCHEMA can ask for — which is the version that still holds when somebody points the kit at their own files.

The result0 patient-data fields in the schema and 0 in the corpus, both asserted before any run may spend — and 0 attack trials. Three of four boundaries checked here hold on evidence, two of them by a gate rather than by reading the code. The one that does not is the one this kit's shape invites: a roster extract is an operational record many hands touch, and whether a note written into one could talk a model out of the rulebook is unmeasured.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted decoys measure whether a model reads the ROSTER to decide what is recorded when the entries point the other way, and whether it follows a credential chain to a second entry — not whether it resists an instruction. The patient-data boundary is checked by two assertions in evals/check_labels.py that must pass before any run may spend: nine patterns over 50 files, and every field name and hint in data/fields.json.

HonestyWhat this does not prove

• Whether an indirect prompt injection in a free-text field could move a call. No attack run exists.
• Whether the kit behaves safely against a hostile provider — a response body crafted to break the list parsing in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed document, and nothing beyond that was tested.
• Whether a real roster extract would carry something this kit mishandles. The corpus has no patient data by construction and the schema asks for none; a real file carries the thing itself, and no part of that path is exercised here.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_credentialing_review fires when the file records at least one encounter whose disposition is "delivered" and whose `coverage` is "not_determined". Both values come out of the same reply; the rule is run afterwards in pure code, over whatever the model returned, and it reads the reply's OWN stated coverage rather than re-deriving it from the reply's facts — so the flag's score is what a downstream consumer of the worksheet would actually route on, error inheritance included.

src/extract.py::compute(), called by extract() on every file, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by identical code. The rulebook lookup it reads lives in a SEPARATE module, src/rulebook.py, which is also what the corpus generator writes gold with — so a policy change and a definition change cannot be made by the same edit.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a licensure determination, a clearance, a block or a notification. Nothing here decides whether anybody is licensed, whether care may or may not be delivered, or whether a booking should stand; nothing is posted, approved or dispatched. coverage and scope_call are proposals with the rulebook's own reasoning attached, and needs_credentialing_review is a routing signal for a person.
• It is NOT a check on whether the extracted facts are right. If the model misreads a register entry and then applies the rulebook correctly to the misreading, the file is flagged — or not flagged — on a wrong fact, and this rule cannot tell. THAT IS NOT HYPOTHETICAL HERE: the run's single error is exactly that shape, and the consistency diagnostic caught 0 of 1.
• It is NOT any organisation's scheduling or credentialing policy. A delivered encounter with an unsettled coverage call is this kit's own simplification, invented for this corpus. No statute, regulation, board rule, compact agreement or health system's policy was consulted, and none is reproduced. A real review weighs what the credentialing system holds outside the file, what the boards concerned require of a remote encounter, and whether the patient's recorded location was right.
• It is NOT a judgement about the rulebook. The flag reads the calls the rulebook produced; if the rulebook is wrong, the flag is confidently wrong with it. The shipped rulebook is illustrative, its jurisdictions and its reciprocity accord are invented, and nobody qualified has reviewed it.

WatchedWhat is watched, and why that one

The kit measures 15 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 8 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 50, extraction_cells 3900, failures 0, refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

extraction · with the model in the path — 3 runs. Columns here are only ever compared with each other.

not a time series No two of these 3 runs measured the same system — they differ on documents, extraction_cells, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

extraction · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 7 chips that all say so.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/extract.py, on any change to data/rulebook.json (which moves the calls the flag reads), on any change to data/fields.json's allowed values for disposition or coverage, and on any corpus regeneration. Changing WHAT GOES TO THE CREDENTIALING OFFICE is a policy change and must be re-scored even though it never changes what the rulebook says.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the rulebook is a JSON file read with json, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold rather than installing a client for a vendor they will never call.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per file — segment, select, prompt, one call, parse, compute — with no branch, no loop and no agent. The two structural differences from its siblings are that the REPLY is a list, and that the document carries TWO per-item families that have to be joined to each other; both change the scorer and the prompt, not the pipeline.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain — the JSON extraction from a fenced reply, the list parsing, the backoff policy, the two section regexes, the prefix matcher and the .env reader are all code somebody has to own.
• No framework means no framework's ecosystem — no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. The reply is a LIST and nothing validates its shape except src/prompt.py::parse, which fails closed to {} rather than to an empty worksheet, because an empty encounters list is a real answer on this kit.
• The provider abstraction covers exactly two wire formats. A third provider is one function and one dict entry, and until somebody writes it the kit runs on two shapes.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-licensure-match on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b000-licensure-match-register, t000-licensure-match-stub recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the rulebook is data read whole into the prompt.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Every clinician file reference, jurisdiction, credential code, encounter reference and date is invented; no real person, board, employer, health system, platform, compact or place is named, and no statute, regulation or board rule is reproduced or named either. ⚠︎ AND THERE IS NO PATIENT DATA OF ANY KIND: an encounter records the jurisdiction the patient was located in, the modality, two dates and a service category, and nothing whatever about the person receiving care. The jurisdiction-match rulebook this kit ships is illustrative and is not an authority. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineWhich encounters reached the worksheet

Did the run list exactly the encounters this file's roster records? Micro precision and recall over encounter references, with the two directions named and never averaged: a MISSED encounter is a recorded one left off, and a PHANTOM encounter is one on the worksheet the roster does not record. Phantoms are broken down by which decoy produced them — an entry cancelled by the schedule amendment, a proposed slot that was never booked, an encounter on another clinician's roster, or a reference that is nowhere in the file. A phantom here is a permission question put in front of a reviewer that nobody ever faced.

MethodHow the test was run

evals/judge.py::score_identification, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, which records the recorded encounters AND the decoy references for every file; evals/check_labels.py asserts before any run may spend that every decoy reference is in its file, collides with no recorded encounter, and that each of the three kinds clears a floor of 8.

How tight can the band be? No tolerance band — a reference is on the worksheet or it is not, compared after whitespace stripping and upper-casing.

Cadence: Re-score on any change to data/rulebook.json's what_is_an_encounter_here, to src/prompt.py's rule 1, or to the corpus's decoy composition.

The decisionWhen to reach for it

In one linePer-(encounter, field) exact match, light normalisation

Does the model's value for each of the fourteen per-encounter fields match gold, after trimming whitespace and punctuation, plus one roster reference per file? THE DENOMINATOR IS GOLD'S ENCOUNTERS, NOT THE RUN'S: a gold encounter the run never returned scores as a miss on all fourteen of its cells rather than disappearing, so a run cannot raise this number by returning less. Two fields are legitimately null — credential_ref and valid_through where the register holds no credential — and a null answer against a null gold is a hit, not an absence.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every run, and the same logic the free floor is scored by — a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same values the file states; evals/check_labels.py asserts every field is populated and in its allowed set on every one of the 275 encounters, and that every stated value — the reference, the jurisdiction, both dates, the credential reference and its valid-through date — appears in the file it labels, before any run is allowed to spend.

How tight can the band be? There is no tolerance band — it is exact match after trimming whitespace and punctuation, never a continuous score to round.

Cadence: Re-score on any change to data/rulebook.json, src/prompt.py, data/fields.json or tools/build_corpus.py. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe two rulebook calls — coverage six-way, scope three-way

Did the run land on the same calls the shipped rulebook produces — six-way for coverage and three-way for scope_call? SCORED ON MATCHED ENCOUNTERS ONLY, and saying so is half of what makes the number honest: a call about an encounter the run never found is an identification failure, and counting it twice would let one mistake sink two graders. The two calls are scored separately because they are asked separately: an active, current, correctly-paired credential can still have a recorded scope that expressly excludes the service, and folding them into one verdict would hide which one a reviewer has to go and check.

MethodHow the test was run

evals/judge.py::score_calls, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's calls are re-derived inside the grader by the same lookup the kit publishes — src/rulebook.py, reading data/rulebook.json — so the truth this grades against can never be a separately-typed label that drifted from the rule.

How tight can the band be? No threshold — each call is one of six or three allowed values, and a reply returning none of them is counted as unanswered rather than folded into a correct cell.

Cadence: Re-run whenever data/rulebook.json or src/rulebook.py changes, whenever the corpus is regenerated, and on any provider or model change.

The decisionWhen to reach for it

In one line⚑ The headline — how often it says the record does not settle it

Of every call this corpus's files do NOT settle, how many did the run settle anyway? That is overconfident, and on a reviewer's worksheet it is the expensive direction: a call recorded as settled is a call nobody re-reads, so an over-confident worksheet quietly removes the reviewer it exists to serve — and the confident value most available here is covered_on_the_record, which reads as a clean bill. Its mirror, overcautious, is counted beside it because saying 'the record does not settle it' where the record does settle it creates work rather than hiding it, and the two costs do not net against each other.

MethodHow the test was run

evals/judge.py::score_calls — the determinacy block, in-process, no key, no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The same rulebook lookup as the grader above, with the rows partitioned by whether GOLD's answer is not_determined. Both directions come out of one pass over the same 550 calls.

How tight can the band be? A rate over a fixed denominator: 110 open calls, 440 settled ones. A run over a different corpus has a different denominator and the two rates are not comparable.

Cadence: Re-run on any change to the rulebook's not_determined branches, to the corpus's unstated-fact composition, or to the provider or model.

The decisionWhen to reach for it

In one lineneeds_credentialing_review against the same rule run over gold

Does the pure-code review decision — an encounter already DELIVERED whose coverage the record does not settle — land on the same files it would land on if every encounter had been read perfectly? It is a BUSINESS CONDITION, so unlike a self-consistency check it genuinely needs labels, and saying so is the point: this is the one number on the page that cannot be computed on unlabelled files.

MethodHow the test was run

evals/judge.py::score_flag, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute(), the same function the run uses, applied to GOLD's own encounters. One rule, one input shape, so a change to the rule moves both sides together and the grader cannot silently grade an old policy.

How tight can the band be? No threshold — two values per encounter and two comparisons.

Cadence: Re-run whenever compute() changes. Changing WHAT GOES TO THE CREDENTIALING OFFICE is a policy change and must be re-scored, even though it never changes what the rulebook says.

The decisionWhen to reach for it