The business caseThe problem this solves

A sanctions screening engine fires an alert whenever a customer's name looks enough like a name on a watchlist. Somebody then has to decide whether the alert is the same party or a coincidence — and the sheet in front of them puts four things next to the answer that look like evidence and are not: the engine's own fuzzy-match score, the two nationalities, the analyst's triage note, and how alike the names read. The answer is in the identifiers, and the identifiers frequently point the opposite way from everything else on the page. The third possibility is the one that gets lost: an alert the file simply does not decide, which is neither a match nor a clearance. Somebody opening each screening alert, reading the customer record beside the watchlist entry it fired on, and deciding whether the two are the same party — which in practice means finding the ONE identifier that settles it and ignoring the four things that look like evidence and are not. It is a few minutes an alert and it is every alert, and the part that goes wrong is not the reading: it is weighing a low fuzzy-match score against a matching passport number, treating a passport number on one record and a national identity number on the other as corroboration, reading a year-only date of birth as agreement, and — most costly of all — reaching for a decision on an alert the file does not decide.

The inputThe actual alert review sheets

The corpus is 50 alert review sheets, 0.03 MB (txt 50). Plain text, one format, invented rather than fetched — and here that is not a convenience, it is the only option. A real screening alert carries a named living person's date of birth, address, nationality and passport number, an allegation about them, and an institution's own view of whether it is true. There is no public corpus of (alert, correct adjudication) pairs and there never will be. Generating it also makes the label mechanical: gold's verdict and deciding identifier are the rulebook lookup over the same eight values the sheet states, never somebody's reading of how alike two records look.

Alert Reference
---------------
ALR-ES-35824

Screening List
--------------
Global Designated Entities Register -- programme: Programme Cindermark
Entry published: 2023-07-01
Engine match score: 0.92

Customer Record
---------------
Name: Tavrick Jarnevik
Date of Birth: 1980-03-02
Place of Birth: Veldt Harbour
Nationality: Corvant
Passport Number: FT5132423

Watchlist Entry
---------------
Listed Name: Tavrick Jarnevik
Also Listed As: T. Jarnevik; Tavrickh Jarnevik
Date of Birth: 1980-03-02
Place of Birth: Veldt Harbour
Nationality: Corvant
Passport Number: FT5132423

Account Status
--------------
live

Analyst Note
------------
Looks like a clear hit to me; the file lines up.

The outcomeWhat a good result looks like

A seventeen-field extracted record per alert, plus one pure-code adjudication taken from eight of those fields: same party, not a match, or — a first-class answer, not a failure — insufficient information, with the identifier that produced the verdict named and, where nothing did, what one extra fact would settle it. A second pure-code rule orders the worklist: an alert that is not dismissible on the file, on an account that is already live, is the one to put in front of a person. Nothing here closes an alert, freezes an account, stops a payment or files anything with anybody.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Working a day's screening alerts before a financial-crime or onboarding desk opens them — the fast tier — 50 of 50 adjudications, 50 of 50 deciding identifiers, 0 false confidences and 0 false clearances on this corpus
  100 pct three-way accuracy against the free analyst-note floor's 30.0 pct, 0 false clearances against the floor's 11 of 35, and 0 false confidences against the floor's 11 of 15.
• Deciding whether the model is worth calling at all, rather than regexing the fields and running the rule in code — run the rule in code first, and use the model for what code cannot read
  The free floor extracts all fifteen structured fields PERFECTLY — 750 of 750 cells — on this corpus's fixed layout. Everything the rulebook needs is regex-reachable, and calling src.rulebook.decide() on the floor's own extracted values would score 100 pct for $0.00. That is a real finding about THIS CORPUS and it is the honest recommendation for anything with a fixed sheet layout.

And where nothing here is good enough:

• Deciding whether this shape is worth buying at all, from these numbers — nothing yet — this corpus cannot answer it
  One run, one tier, one seed, 50 alerts, everything perfect. Nothing here separates any two configurations, and a corpus nothing fails cannot rank a model. What the numbers DO support is narrower and real: adjudicating from the analyst's note is measurably worse than useless on this shape of decision, and a self-consistency check catches all 35 of that shortcut's errors with no labels at all.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-sanction-screen. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts on 127.0.0.1:8052, every route answers (/, /app.js, /app.css, /api/fields, /api/rulebook, /api/doc), the 50-alert picker populates, the field table draws its seventeen empty rows and the shipped rulebook renders in full from data/rulebook.json; clicking Adjudicate returns the no-key sentence rather than a stack trace, and the escalation row reads “not computed — one of the two values the rule needs was missing” rather than “no”. python -m evals.check_labels and python -m evals.run --baseline both pass with no network access at all.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per screening alert, measured around the adapter call under 12 concurrent workers

Current processWhat it replaces

Somebody opening each screening alert, reading the customer record beside the watchlist entry it fired on, and deciding whether the two are the same party — which in practice means finding the ONE identifier that settles it and ignoring the four things that look like evidence and are not. It is a few minutes an alert and it is every alert, and the part that goes wrong is not the reading: it is weighing a low fuzzy-match score against a matching passport number, treating a passport number on one record and a national identity number on the other as corroboration, reading a year-only date of birth as agreement, and — most costly of all — reaching for a decision on an alert the file does not decide.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 0 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings; an alert with none falls back to one whole-document segment

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 50 alerts cut into 300 sections in two thousandths of a second, in process, with no model and no network.

LicenceLicence

MIT — this repository's own licence. EVERY name, watchlist name, programme name, country, place of birth and identifier in this corpus is INVENTED: person names are assembled from syllable pools written for the generator, countries and places are made up, list and programme names are made up, and passport, national-identity and tax-reference numbers are generated from made-up formats. No real sanctions programme, designation list, authority, jurisdiction, country-specific regime, institution or person is named, referenced or alluded to anywhere in this kit. The adjudication rulebook this kit ships (data/rulebook.json) is ILLUSTRATIVE and is not an authority: it reproduces no programme, list, supervisor's guidance, industry matching standard or screening-vendor tuning.

Bring your ownBring your own alert review sheets

REPLACE data/rulebook.json FIRST. It is this kit's own construction, your institution's policy about identifier strength is not this one, and everything downstream reads it: src/rulebook.py loads it, src/prompt.py renders it into every call, and tools/build_corpus.py writes gold with it. Then replace data/corpus/*.txt, write your own data/fields.json, and supply a gold record per alert. SECTION_HINTS in src/select.py maps fields to section headings and will need editing for a different sheet layout; when it does not match, selection falls back to the whole document — slower, more expensive, always correct. compute() in src/extract.py is this kit's own escalation rule and is the second thing to replace.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 3,105 tokens assembled, 1,228 are contexts — 40% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

The first three parts were measured token-for-token via evals/prompt_tokens.py's nested-prefix method — four calls at max_tokens=1, each part's size the difference between two consecutive prompt_tokens counts the provider itself returned. ⚑ IT WAS RUN ON TWO DIFFERENT ALERTS AND RETURNED 972 / 973 / 905 BOTH TIMES (kits/UC0052-sanction-screen/results/tokens-p001-sanction-screen.json on SCR-0001 and tokens-p002-sanction-screen.json on SCR-0038), which is the evidence that the 2,850-token fixed prefix really is fixed and is paid on every one of the 50 calls. The fourth part is DERIVED rather than separately measured: the worked example's own call reported 3,105 input tokens, so its alert sections are 3,105 − 2,850 = 255. That subtraction is exact only because the prefix was measured twice and did not move.

Full promptThe prompt in full

You read a SANCTIONS SCREENING ALERT REVIEW SHEET -- one customer record beside the watchlist entry a screening engine matched it to -- and extract structured fields from it. You return JSON and nothing else.

You are PROPOSING an adjudication for a human to decide. You clear nothing, you block nothing and you file nothing: no alert is closed, no party is designated or de-designated, no account is frozen or released, no payment is stopped or let through, and no report is made to anybody.

RULES, in order of importance:
1. If the sheet does not state a field, return null for it. Do not infer it and do not use what you know about the world. Every name, list, programme, place and identifier on these sheets is invented for this exercise -- there is nothing to recognise.
2. `verdict` and `deciding_identifier` are decided ONLY by the ADJUDICATION RULEBOOK given below, applied to eight values -- the two identifier types, the two identifier values, the two dates of birth and the two places of birth. Work through these five checks IN ORDER and STOP at the first one that fires:
   a. STRONG IDENTIFIER, SAME VALUE. If BOTH records carry a strong identifier OF THE SAME TYPE and the two values are the same, answer 'same_party' and name that identifier. This outranks everything below it.
   b. STRONG IDENTIFIER, DIFFERENT VALUE. If both carry a strong identifier of the same type and the values differ, answer 'not_a_match' and name that identifier. This also outranks everything below it.
   c. MODERATE CONFLICT. Otherwise, take the moderate identifiers that are COMPARABLE on both records. If any of them DISAGREES, answer 'not_a_match' and name it. Read date of birth before place of birth.
   d. MODERATE AGREEMENT. Otherwise, if at least 2 comparable moderate identifiers AGREE and none disagrees, answer 'same_party'. Name 'date_of_birth_and_place_of_birth' when both agree.
   e. OTHERWISE answer 'insufficient_information' and set deciding_identifier to 'none'.
3. TWO STRONG IDENTIFIERS OF DIFFERENT TYPES ARE NOT COMPARABLE. A passport number on the customer record and a national identity number on the watchlist entry say NOTHING about each other. Do not treat that as weak evidence either way -- checks (a) and (b) simply do not fire, and the rule falls through to the moderate identifiers as though neither record carried one.
4. A PARTIAL DATE OF BIRTH IS NOT COMPARABLE. A date is comparable only when BOTH records carry a FULL calendar date. A year-only date ('1978') or a year-and-month date ('1978-04') is a stated fact and is still not comparable -- it neither agrees nor disagrees with anything, and it contributes nothing in either direction. Copy it verbatim into the field and do not pad it out.
5. NAMES, NATIONALITIES AND THE ANALYST'S NOTE ARE FIELDS TO COPY, NOT EVIDENCE. The alert exists because the names looked alike, so name similarity settles nothing. A watchlist record's nationality is routinely stale or secondary and disagrees for the same party all the time. The analyst's note is one person's impression written before anybody compared the identifiers, and on these sheets it often points the wrong way. None of the three is an input to the verdict.
6. `insufficient_information` IS A REAL ANSWER AND YOU SHOULD REACH FOR IT WHEN THE FILE DOES NOT DECIDE. It means these two records do not carry enough to be separated or joined. It is not a clearance and it is not a match, and guessing one of the other two instead is the worst thing you can do on this sheet.
7. Copy names, places, nationalities and identifier values verbatim from the sheet.
8. Use the exact allowed value for a field that lists them. 'none' is a VALUE for the two identifier-type fields, not a blank.
9. Return every field named in the schema, even when the answer is null.

ADJUDICATION RULEBOOK (the authority for `verdict` and `deciding_identifier`; this is an ILLUSTRATIVE rulebook shipped with this kit, not any real list, programme, supervisor's guidance or vendor tuning)

IDENTIFIER STRENGTH TIERS
  strong:    passport_number, national_id_number, tax_reference
  moderate:  date_of_birth, place_of_birth
  weak:      nationality, name_similarity, engine_match_score, analyst_note

WHAT EACH TIER DOES
  strong    A strong identifier is a number issued to ONE party and to no other. In this rulebook a strong identifier that is present on both records SETTLES the alert, in whichever direction it points, and it outranks every disagreement below it. Two records carrying the same passport number are the same party even if the names are spelled differently and the nationalities disagree; two records carrying different passport numbers are different parties even if the name and the full date of birth agree exactly. That is the sharpest claim in this file and it is deliberately absolute.
  moderate  A moderate identifier narrows a population; it does not name a party. This rulebook needs MIN_MODERATE_AGREEMENTS of them, agreeing and comparable, before it will join two records — and ONE of them conflicting is enough to separate them. Weak agreements never add up to a strong identifier, and no number of them substitutes for one.
  weak      A weak field is RECORDED, is EXTRACTED, and is never an input to the verdict. Nationality on a list record is often stale or secondary; the screening engine's own match score is a statement about strings; and the analyst's note is one person's impression written before anybody compared the identifiers. All three are on the alert, all three are read back, and none of them moves the answer.

WHEN A MODERATE IDENTIFIER IS EVEN COMPARABLE
  date_of_birth:   A date is comparable only when BOTH records carry a FULL calendar date. A year-only or year-and-month date is a STATED FACT and not a missing one — and it is still not comparable, because it cannot agree or disagree with anything at the precision the rule needs. A partial date contributes nothing in either direction.
  place_of_birth:  A place is comparable only when both records name one. A place published on one record and absent from the other contributes nothing.

IDENTIFIERS OF DIFFERENT TYPES
  TWO STRONG IDENTIFIERS OF DIFFERENT TYPES ARE NOT COMPARABLE. A passport number on one record and a national identity number on the other tell you nothing about each other, and this rulebook does not pretend otherwise — it falls through to the moderate fields exactly as though neither record carried an identifier at all. That is one of the two commonest routes to insufficient_information on this corpus.

DECISION ORDER -- stop at the first check that fires
  1. STRONG IDENTIFIER, SAME VALUE. If both records carry a strong identifier OF THE SAME TYPE and the values match, answer same_party. Nothing below this line can overturn it.
  2. STRONG IDENTIFIER, DIFFERENT VALUE. If both records carry a strong identifier of the same type and the values differ, answer not_a_match. Nothing below this line can overturn it either.
  3. MODERATE CONFLICT. With no strong identifier comparable, any single comparable moderate identifier that DISAGREES answers not_a_match. Date of birth is read before place of birth.
  4. MODERATE AGREEMENT. At least min_moderate_agreements comparable moderate identifiers agreeing, and none disagreeing, answers same_party.
  5. OTHERWISE. Answer insufficient_information, and name what would settle it.

MINIMUM MODERATE AGREEMENTS TO JOIN TWO RECORDS: 2

NAMING THE DECIDING IDENTIFIER
  Every verdict names the identifier that produced it, and `none` is the honest answer for insufficient_information. A verdict with no named identifier behind it is an opinion; a verdict that names the wrong one is right by accident, which is why this kit scores the two separately.

WHAT THIS RULEBOOK IS NOT
  This kit PROPOSES an adjudication with its reasoning and names what it could not determine. It clears nothing, blocks nothing and files nothing: no alert is closed, no party is designated or de-designated, no account is frozen or released, no payment is stopped or let through, and no report of any kind is made to anybody. A human makes the call.

Extract these fields:
- alert_id (string) -- the alert reference, verbatim
- customer_name (string) -- the name on the CUSTOMER RECORD, verbatim
- listed_name (string) -- the PRIMARY listed name on the WATCHLIST ENTRY, verbatim -- the 'Listed Name' line, never an alias from the 'Also Listed As' line
- customer_identifier_type (enum) one of: passport_number, national_id_number, tax_reference, none -- which strong identifier the CUSTOMER RECORD carries. Return 'none' when the record says no secondary identifier is on file -- 'none' is a value, never a blank
- customer_identifier_value (string) -- the CUSTOMER RECORD's value for that identifier, verbatim. Return null when customer_identifier_type is 'none'
- listed_identifier_type (enum) one of: passport_number, national_id_number, tax_reference, none -- which strong identifier the WATCHLIST ENTRY carries. Return 'none' when the entry says no identifier is published -- it may be a DIFFERENT type from the customer record's, and that is a real and common state
- listed_identifier_value (string) -- the WATCHLIST ENTRY's value for that identifier, verbatim. Return null when listed_identifier_type is 'none'
- customer_dob (string) -- the CUSTOMER RECORD's date of birth EXACTLY as stated, including a partial one -- copy '1978' or '1978-04' as it stands and do not pad it. Return null only where the record says the date of birth is not recorded
- listed_dob (string) -- the WATCHLIST ENTRY's date of birth EXACTLY as stated, including a partial one. Return null only where the entry says no date of birth is published
- customer_place_of_birth (string) -- the CUSTOMER RECORD's place of birth, verbatim. Return null where the record says it is not recorded
- listed_place_of_birth (string) -- the WATCHLIST ENTRY's place of birth, verbatim. Return null where the entry says none is published
- customer_nationality (string) -- the nationality on the CUSTOMER RECORD, verbatim. Report it as stated -- it is NOT an input to the verdict
- listed_nationality (string) -- the nationality on the WATCHLIST ENTRY, verbatim. Report it as stated -- it is NOT an input to the verdict, and a list record's nationality is often stale or secondary
- account_status (enum) one of: pending_onboarding, live -- is the customer's account already live, or is onboarding still pending?
- analyst_note (string) -- the triaging analyst's own free-text remark on the alert, copied verbatim
- verdict (enum) one of: same_party, not_a_match, insufficient_information -- the adjudication this alert's own identifiers produce. Decide it STRICTLY from the two identifier types and values, the two dates of birth and the two places of birth -- never from the names, never from the nationalities and never from the analyst's note. Work through the five checks IN ORDER, stopping at the first that fires. (1) Both records carry a strong identifier OF THE SAME TYPE with the SAME value: 'same_party'. (2) Both carry one of the same type with DIFFERENT values: 'not_a_match'. (3) Otherwise, any comparable moderate identifier that DISAGREES: 'not_a_match'. (4) Otherwise, at least two comparable moderate identifiers agreeing and none disagreeing: 'same_party'. (5) Otherwise: 'insufficient_information'
- deciding_identifier (enum) one of: passport_number, national_id_number, tax_reference, date_of_birth, place_of_birth, date_of_birth_and_place_of_birth, none -- WHICH identifier produced that verdict. A strong identifier that matched or conflicted is named on its own. A single conflicting moderate identifier is named on its own. Two agreeing moderate identifiers are 'date_of_birth_and_place_of_birth'. When the verdict is 'insufficient_information' the answer is 'none' -- nothing on the file decided it

Return a JSON object with exactly these keys: alert_id, customer_name, listed_name, customer_identifier_type, customer_identifier_value, listed_identifier_type, listed_identifier_value, customer_dob, listed_dob, customer_place_of_birth, listed_place_of_birth, customer_nationality, listed_nationality, account_status, analyst_note, verdict, deciding_identifier
Use null for any field the sheet does not state.

ALERT REVIEW SHEET
------------------
Alert Reference
---------------
ALR-SL-46955

Customer Record
---------------
Name: Nevrid Thackmere
Date of Birth: 1966-12-28
Place of Birth: Fennmarch
Nationality: Kastelia
Passport Number: YP5276569

Watchlist Entry
---------------
Listed Name: Nevrid Thackmere
Also Listed As: N. Thackmere; Naevrid Thackmere
Date of Birth: 1966-12-28
Place of Birth: Fennmarch
Nationality: Kastelia
Passport Number: MW2707565

Account Status
--------------
live

Analyst Note
------------
I am satisfied these are the same party.

Raw responseThe raw response

{"alert_id":"ALR-SL-46955","customer_name":"Nevrid Thackmere","listed_name":"Nevrid Thackmere","customer_identifier_type":"passport_number","customer_identifier_value":"YP5276569","listed_identifier_type":"passport_number","listed_identifier_value":"MW2707565","customer_dob":"1966-12-28","listed_dob":"1966-12-28","customer_place_of_birth":"Fennmarch","listed_place_of_birth":"Fennmarch","customer_nationality":"Kastelia","listed_nationality":"Kastelia","account_status":"live","analyst_note":"I am satisfied these are the same party.","verdict":"not_a_match","deciding_identifier":"passport_number"}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check a bank's sanctions alerts against the watchlist — 50 alert review sheets. One model answered, and every answer was then graded Four different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-18. The provider that actually ran this run publishes no rate card this repo commits, so nothing here is what was actually paid — the real spend for this kit's build is recorded in the commit history and the shared call ledger, not on this page.

The gradersFour ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

The analyst-note floor (evals/baseline.py) and the scorer (evals/judge.py) are both pure code and cost $0.00 to run against any result set. The figure above is the ONE scored run's own token counts (50 alerts) priced at Google Gemini 3 Flash's published rate — the same basis cost_per_query_usd uses. ⚠︎ IT IS NOT WHAT BUILDING THIS KIT COST. Nine further paid calls sit outside it and are committed in results/: two prompt-split measurements of four calls each at max_tokens=1 (tokens-p001 and tokens-p002, $0.0089 together) and one worked-example replay (example-SCR-0006, $0.0026). All 59 live calls together project to $0.1654 on this card.

Cost driversWhat actually moves the bill

• The fixed prefix — system prompt (972 tokens), adjudication rulebook (973) and field schema (905) — is 2,850 of 3,105 tokens on the worked example, 92 pct, and is byte-identical on every one of the 50 calls. The alert itself is 255 tokens. The bill is the policy and the instructions, not the alert.
• Provider-side reasoning. 14,486 of the run's 25,391 output tokens (57.1 pct) were reasoning rather than the JSON record, on a seventeen-key reply that is about 220 tokens of actual content. Reasoning was left at the provider's default and nothing here has measured what turning it off would cost in accuracy.
• Output length: the model returns a full seventeen-key JSON record every call, including the analyst's note copied back verbatim, whatever the alert says. Seventeen fields is a lot for a decision that turns on eight of them, and four of the seventeen are decoys the rule never reads.

Your volumeWhat it costs at your volume

Linear in alerts: each call is independent, carries the same fixed prefix and shares nothing with its neighbours, so 500 alerts cost ten times 50. WALL CLOCK IS NOT LINEAR, because this harness is concurrent — 50 alerts took 20.9 seconds at 12 workers, and EVAL_WORKERS is the knob rather than a rewrite. ⚠︎ AND THE RULEBOOK DOES NOT SCALE LINEARLY, IT SCALES WORSE: it is sent whole in every call, so a real institution's matching policy — dozens of identifier types with jurisdiction-specific comparability rules rather than three tiers and two moderate fields — multiplies the input side of every call at once. At that size the comparison has to move into code before the call, with the model reading fields and nothing else.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 50 screening alert review sheets, one completion call each, on one tier.

The codeEvery file, and what it is for

Six modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the alert review sheet into addressable sections, pure code — 6 per alert, 300 across the corpus

# Cut a document into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

pick which sections carry each field, pure code — the verdict and the deciding identifier are mapped to the two RECORDS and nothing else, never to the analyst's note and never to the nationalities, and the whole Screening List section (the list, the programme, the publication date and THE ENGINE'S OWN MATCH SCORE) is mapped by nothing and never sent

You change it to: map fields to your own alert sheet's headings; unmatched falls back to the whole document. It is also where you decide what NEVER reaches the model — this kit withholds the engine's own match score there, deliberately

# Pick which sections plausibly carry each field. Pure code -- the last deterministic step before

SECTION_HINTS = {
def for_field(secs, field):
def plan(secs, fields):

src/rulebook.pyrulebook

the adjudication rule, loaded from data/rulebook.json — identifier strength tiers, a stopping order in which a strong identifier outranks everything below it, an explicit definition of when a moderate field is even COMPARABLE, and a third verdict for the case the file does not decide. Data, not code, so a forker can open it and disagree with it

# THE ADJUDICATION RULEBOOK, loaded from data/rulebook.json. Pure code, no model, no network.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
RULEBOOK_PATH = os.path.join(HERE, "data", "rulebook.json")
def load(path=RULEBOOK_PATH):
R = load()
VERDICTS = tuple(R["verdicts"])
STRONG = list(R["identifier_tiers"]["strong"])
MODERATE = list(R["identifier_tiers"]["moderate"])
WEAK = list(R["identifier_tiers"]["weak"])
MIN_MODERATE = int(R["min_moderate_agreements"])

src/prompt.pyprompt

assemble one call for all seventeen fields, with the rulebook RENDERED into it from data/rulebook.json rather than restated in prose — a policy the model has never been shown is not a policy

# Assemble the alert-adjudication prompt. One prompt per alert, all seventeen fields in it.

SYSTEM = (
def rulebook_block(r=None):
RULEBOOK_TEXT = rulebook_block()
def field_schema(fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/extract.pyextract — a swap seam

the AI layer, one provider one key — plus the pure-code business-condition check downstream: an alert that is not dismissible on the file, on an account that is already live, is escalated to a person

You change it to: the escalation rule itself — this kit ships two values (not dismissible AND the account is live) and a real desk weighs the exposure, the customer's own risk rating, how long the alert has been open and who is allowed to decide it. It is deliberately NOT the same function as the rulebook lookup, so changing WHAT JUMPS THE QUEUE does not change WHAT COUNTS AS THE SAME PARTY

# Extract one screening alert's fields: segment, select, prompt, one model call, then a pure-code

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 4000
def load_fields():
def load_doc(alert_id):
def documents():
def decide(values):
def correct_verdict(values):

evals/judge.pyjudge

score field accuracy, the three-way verdict AND its collapse onto 'does this alert stay open', the FALSE-CONFIDENCE count in both directions, the deciding identifier, and the escalation flag, all separately, pure code

# Score a screening-alert adjudication run. PURE CODE -- gold is exact and the answer is one value

def norm(v):
def equal(field, got, want):
NULLABLE = ("customer_identifier_value", "listed_identifier_value", "customer_dob", "listed_dob",
def score(fields, records, golds):
def _matrix(rows, positive):
def score_verdicts(records, flags, golds):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3109 input and 507 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's alert review sheets are entirely synthetic (tools/build_corpus.py, seed 20260822): no real person, list, programme, authority, jurisdiction, institution or alert exists in the corpus, and nothing was fetched from anywhere. Every person name is assembled from an invented syllable pool, every country and place is invented, every list and programme name is invented, and every passport, identity and tax number is generated from a made-up format and is valid in no scheme. The only outbound traffic the kit makes is one chat-completion request per alert to the configured provider, carrying the mapped sections of one alert plus the fixed prompt and the shipped rulebook. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only. ⚠︎ A REAL DEPLOYMENT OF THIS SHAPE WOULD SEND SPECIAL-CATEGORY PERSONAL DATA TO A THIRD-PARTY MODEL PROVIDER — a named person's date and place of birth, nationality, identity-document number and an allegation about them — and none of that path is exercised or reasoned about anywhere in this kit.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser.

The experimentWe did not attack it — and three of four boundaries hold

The three boundaries that hold were confirmed by reading the code, not by a run: no code path closes an alert, freezes an account or files anything; the rulebook lookup and the escalation rule read only enums, identifier values, dates and places, so the analyst's note cannot reach either; and the key and base URL are redacted out of any error before it is returned. The fourth is open. An indirect prompt injection needs a field an outside party authored, and this kit has exactly one — the analyst's triage note — which makes it the obvious place to attack and the reason the gap is named rather than glossed. Confirmed by reading the code, not by a run, on 2026-08-22 — no attack run was fired; see redteam.why.

The first three boundaries hold, confirmed by reading the code rather than by an attack run. The fourth is open and is named as open — an indirect prompt injection in the analyst's note is exactly the attack this kit's shape invites, and nothing here has tried it.

The result0 attack trials, and three of four boundaries checked here hold on evidence. The one that does not is the one this kit's own shape invites: the analyst's triage note is free text a third party wrote, and whether an instruction hidden in it could move an adjudication next to a sanctions queue is unmeasured.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted ambiguity is a REGISTER mismatch — a confident note on an alert the identifiers separate, a dismissive one on an alert they join — which measures whether a model compares identifiers when the prose points the other way. It does not measure whether a model obeys an instruction hidden in the same field. Those are different failures and only one of them is measured here.

HonestyWhat this does not prove

• Whether an indirect prompt injection in the analyst_note field could move an adjudication. No attack run exists.
• Whether the kit behaves safely against a hostile provider — a response body crafted to break the JSON extraction in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed document, and nothing beyond that was tested.
• Anything at all about handling real screening data. The corpus has no personal data by construction; a real alert carries a named living person's date and place of birth, nationality, identity-document number and an allegation about them, and none of that path — retention, provider terms, jurisdiction of processing, or who may read a result file — is exercised or reasoned about anywhere in this kit.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_escalation fires when the alert is NOT dismissible on the file AND the customer's account is already live — verdict != "not_a_match" and account_status == "live". Both values come out of the same reply; the rule is run afterwards in pure code, over whatever the model returned, and never over gold. `insufficient_information` counts as not-dismissible, deliberately and emphatically: an alert nobody can decide, on an account that is already running, is precisely the one to put in front of a human, and treating “cannot tell” as a clearance is the single worst thing anything in this kit could do. A reply missing either value returns None rather than False — an unknown is not a pass.

src/extract.py::compute(), called by extract() on every alert, by the local app on every click, and by evals/baseline.py on the free floor's own output so the two are routed by identical code. The adjudication it reads lives in a SEPARATE module, src/rulebook.py::decide(), which is also what the corpus generator wrote gold with and what src/prompt.py states to the model — one definition, three readers.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a screening decision, a clearance or a block. Nothing here closes an alert, designates or de-designates a party, freezes or releases an account, stops or releases a payment, or files a report with anybody, and verdict is a proposal with the deciding identifier attached rather than a decision anybody is bound by. A human makes the call.
• It is NOT a screening system. It runs no matching engine, subscribes to no list, polls nothing, schedules nothing and knows nothing about why an entry was published. It reads one alert review sheet that somebody else assembled.
• It is NOT a check on whether the extracted values are right. If the model misreads an identifier or a date and then reasons correctly from the misreading, the alert is escalated (or not escalated) on a wrong fact and this rule cannot tell. The consistency diagnostic in evals/judge.py is the closest thing to that check, it is reported separately, and it is blind to exactly the same case.
• It is NOT anybody's real alert-handling policy. Not-dismissible-and-live is this kit's own simplification, invented for this corpus. No supervisor's guidance, institution's procedure or regulatory rule was consulted, and none is reproduced. A real desk weighs the exposure, the customer's own risk rating, how long the alert has been open and who is allowed to decide it.
• It is NOT a judgement about the rulebook. The flag reads the verdict the rulebook produced; if the rulebook is wrong, the flag is confidently wrong with it. The shipped rulebook is illustrative and is not an authority.

WatchedWhat is watched, and why that one

The kit measures 16 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 9 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (documents 50, extraction_cells 850, failures 0, refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

3 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

extraction · with the model in the path — 2 runs. Columns here are only ever compared with each other.

not a time series No two of these 2 runs measured the same system — they differ on provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

extraction · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 7 chips that all say so.

DeviationsWhat deviated

0 breaches across 3 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() in src/extract.py, on any change to data/rulebook.json (which moves the verdict the flag reads), on any change to data/fields.json's allowed values for verdict or account_status, and on any corpus regeneration. Changing WHAT JUMPS THE QUEUE is a policy change and must be re-scored, even though it never changes what counts as the same party.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in-process, the rulebook is a JSON file read with json, the provider is reached over urllib, and the UI is one HTML file with no build step. A forker runs this on whichever key they already hold, with one clone and no install.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per alert — segment, select, prompt, one call, parse, adjudicate, compute — with no branch, no loop and no agent. Anything that looks like orchestration in a kit this size is a diagram of a straight line.

The other sideWhat a framework costs you

• Everything is hand-rolled, so everything is yours to maintain. The JSON extraction from a fenced reply, the backoff policy, the section regex and the .env reader are all code somebody has to own — and a sibling kit in this series has a concrete bill for exactly that, having lost two documents of a paid run to a retry policy that only covered HTTP status codes.
• No framework means no framework's ecosystem — no tracing, no eval harness beyond the one in evals/, no prompt registry, no schema validation library. What ships is what is in the repository.
• The provider abstraction covers exactly two wire formats. A third provider is one function and one dict entry, and until somebody writes it the kit runs on two shapes.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-sanction-screen on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 2 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. Only one run has recorded a call latency; the other 2 runs on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. EVERY name, watchlist name, programme name, country, place of birth and identifier in this corpus is INVENTED: person names are assembled from syllable pools written for the generator, countries and places are made up, list and programme names are made up, and passport, national-identity and tax-reference numbers are generated from made-up formats. No real sanctions programme, designation list, authority, jurisdiction, country-specific regime, institution or person is named, referenced or alluded to anywhere in this kit. The adjudication rulebook this kit ships (data/rulebook.json) is ILLUSTRATIVE and is not an authority: it reproduces no programme, list, supervisor's guidance, industry matching standard or screening-vendor tuning. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-field exact match, light normalisation

Does the model's value for each of the seventeen fields match gold, after trimming whitespace and punctuation? Six fields are legitimately null — an identifier value where its type is none, and a date or place of birth the record does not carry — so a null there is a hit, not a miss, when gold agrees. ⚠︎ A PARTIAL DATE IS NOT ONE OF THOSE: “1978” is a stated value, and returning null for it is a miss.

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every call, and the same field-match logic the free floor is scored by — a floor and a real run scored by two different scorers cannot be compared honestly.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same values the sheet states; evals/check_labels.py asserts every non-nullable field is populated on every row, that each identifier value's nullness agrees with its own type in BOTH directions, that every identifier type is one the rulebook carries, and that each of the five hard cases clears its own floor, before any run is allowed to spend.

How tight can the band be? There is no tolerance band — it is exact match after trimming whitespace and punctuation, never a continuous score to round.

Cadence: Re-score on any change to data/rulebook.json, src/prompt.py, data/fields.json or tools/build_corpus.py — the first three change what is asked, the fourth changes what is asked about. Re-run evals/run.py (paid) on any change to MAX_TOKENS or the provider/model.

The decisionWhen to reach for it

In one lineThe adjudication, three-way and collapsed — with the false-confidence count

Three questions at once. Three-way: did the run land on the same verdict the rulebook produces — same_party, not_a_match or insufficient_information? Collapsed: OF EVERY ALERT THAT MUST STAY OPEN — anything that is not a clean not_a_match — how many did the run leave open? The expensive direction is a false_clearance, an alert the rulebook would have kept and the run closed, because a closed alert is never looked at again. AND, NAMED ON ITS OWN, THE NUMBER THIS KIT EXISTS TO PUBLISH: of the alerts the file CANNOT DECIDE, how many did the run answer with a decision anyway? That is false_confidence, and its opposite — hiding behind “I cannot tell” on an alert the file does decide — is published beside it as false_caution, because publishing one direction alone would flatter the kit.

MethodHow the test was run

evals/judge.py::score_verdicts, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's verdict is re-derived inside the grader by the same lookup the kit publishes — src/rulebook.py::decide(), reading data/rulebook.json — so the truth this matrix grades against can never be a separately-typed label that drifted from the rule.

How tight can the band be? No threshold — the verdict is one of three allowed values, and a reply returning none of them is counted as unanswered rather than folded into a correct cell. On a screening queue that distinction is the whole difference between an alert nobody adjudicated and an alert somebody dismissed.

Cadence: Re-run whenever data/rulebook.json or src/rulebook.py changes, whenever the corpus is regenerated, and on any provider or model change.

The decisionWhen to reach for it

In one lineWhich identifier produced the verdict

Did the run name the same identifier the rulebook's own lookup names as having settled the alert — one of the three strong identifiers, the date of birth, the place of birth, the two together, or none where nothing on the file decided it? Scored SEPARATELY from the verdict, because a run can reach the right answer for the wrong reason, and on an adjudication a person has to sign, the reason is half of it. right_verdict_wrong_reason counts exactly that case.

MethodHow the test was run

evals/judge.py::score_verdicts, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/rulebook.py::decide()'s own deciding_identifier, re-derived inside the grader from gold's values — never a separately-typed label.

How tight can the band be? No threshold — seven allowed values and a comparison. Gold uses five of the seven (the three strong types, the two-moderate pair, the two single moderates, and none), and evals/check_labels.py fails the corpus if fewer than four distinct values appear, because a seven-value grader with two live classes is a smaller grader wearing a bigger label.

Cadence: Re-run whenever data/rulebook.json changes — the strength tiers and the stopping order both move this answer.

The decisionWhen to reach for it

In one lineneeds_escalation against the same rule run over gold

Does the pure-code escalation decision — not dismissible on the file AND the account is already live — land on the same alerts it would land on if both fields had been read perfectly? It is a BUSINESS CONDITION, so unlike a self-consistency check it genuinely needs labels, and saying so is half of what makes the number believable.

MethodHow the test was run

evals/judge.py::score_verdicts, in-process, no key and no model.

Every grader on these pages scored the same 50 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/extract.py::compute(), the same function the run uses, applied to GOLD's verdict and account_status. One rule, two inputs, so a change to the rule moves both sides together and the grader cannot silently grade an old policy.

How tight can the band be? No threshold — two values and a comparison.

Cadence: Re-run whenever compute() changes. Changing WHAT JUMPS THE QUEUE is a policy change and must be re-scored, even though it never changes what counts as the same party.

The decisionWhen to reach for it