The business caseThe problem this solves

Before a document leaves an organisation — a support ticket forwarded to a vendor, a lease notice attached to an email, a claim form logged in a ticketing system — someone has to find and mask every span of personal data in it. Today that is a person reading the document and blacking out or replacing each value by hand, one document at a time, or it is not done at all. Someone reading a document line by line and manually finding, then blacking out or replacing, every span of personal data in it — names, SSNs, emails, phone numbers, addresses, dates of birth, card numbers — before it can be shared or filed externally.

The inputThe actual documents

The corpus is 18 documents, 0.01 MB (txt 18). Self-authored on purpose, and named as the one deliberate exception to how the other kits in this repo build corpora: this kit's whole property under test is whether a detector finds real people's sensitive data, so testing it against real people's data is the one thing it must never do (data/corpus/SOURCES.md). 18 document 'kinds' — HR letters, support and cancellation emails, a medical intake form, insurance correspondence, an invoice, a lease notice, a mortgage application, a jury summons, a background-check report and more — each with 3-7 fabricated spans, so the detector is tested against a variety of house styles and format variants (dashed vs. run-together SSNs, four DOB spellings, three CARD groupings) rather than one template with the names swapped.

RIVERSTONE APARTMENTS — Lease Renewal Notice

Tenant: Ingrid Solberg (DOB 1990-03-22)
Unit Address: 88 Fenwick Row, Unit 4, Minneapolis, MN 55401

Your current lease expires in 60 days. To renew, confirm via ingrid.solberg@leasehub.net or call the leasing office at 612-555-0187.

We look forward to another year.

The outcomeWhat a good result looks like

You pick a document and get two views of the same result: a redacted document with every detected span replaced by [CATEGORY], and a highlighted view showing exactly which text was flagged. On the run behind these pages (r002, the fast tier, thinking off), 87 of 88 labelled spans were caught, with precision and recall both at 0.9886; the reasoning tier (r003) reached 1.0 recall at 0.9888 precision on the same corpus.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• The organisation needs zero disclosures and can tolerate some over-masking — the reasoning tier (r003)
  0 leaked spans of 88 — the only run on record with a perfect leak rate — at the cost of the same single over-redaction flash also made and roughly 1.5x the latency (p50 2179ms vs 1476ms).
• Latency or cost matters more than the last percentage point of recall — the fast tier (r002)
  87 of 88 caught at roughly 4-5x lower latency than the same model with thinking left on, and a fraction of the reasoning tier's per-query cost on the illustrative rate card (see Cost.cost_by_model).
• A forker is deciding whether to disable thinking at all — always, for this task
  r001 vs r002 is the same model, same prompt, same corpus, with exactly one variable changed — recall rose from 0.6932 to 0.9886 and 4 fewer documents failed outright.
• Only SSN, EMAIL, PHONE and CARD need catching, never NAME or ADDRESS — the free regex baseline (evals/baseline.py), or neither
  It is perfect on PHONE, SSN and CARD and 9 of 11 on DOB, for zero cost and zero latency — a model buys almost nothing on these four categories.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-07. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-07 — r002-docs-redact. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Verified from a cold clone with no key configured: python -m tools.build_corpus renders the same 18 documents and 88 labelled spans in 0.03 seconds (deterministic — re-running it reproduces the files already committed byte-for-byte), and python -m evals.check_labels confirms all 88 span texts are exact substrings of their documents. Nothing about the detection numbers can be reproduced without a key — this kit calls the model once per document and keeps nothing between runs.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. the model call only, one per document, thinking explicitly disabled, 18 documents run serially.

Current processWhat it replaces

Someone reading a document line by line and manually finding, then blacking out or replacing, every span of personal data in it — names, SSNs, emails, phone numbers, addresses, dates of birth, card numbers — before it can be shared or filed externally.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one document is

none — a document is a few hundred words and goes into the one call whole; there is no segment/select layer (see src/prompt.py's module docstring).

SetupWhat the setup figure measured

There is no index. tools/build_corpus.py renders the 18 documents and their labels from one table of fabricated entities — the only preparation step, pure code, no model, no key. Measured by re-running it: 18 documents, 88 labelled spans, in 0.03 seconds, byte-identical to what was already on disk.

LicenceLicence

MIT, same as the rest of the repo. No third-party content, no scraped source, nothing to attribute — every name, SSN, DOB, email, phone, address and card number is fabricated, never sourced from a real individual.

Bring your ownBring your own documents

Replace data/corpus/*.txt, write your own data/labelled.jsonl (one line per document: {"doc": "<filename>", "spans": [{"text": "<exact substring>", "category": "..."}]}), and run evals/check_labels.py before spending anything — it fails loudly on a label whose text is not a literal substring of its document. data/categories.json is the schema the prompt is generated from; add an eighth category there and nowhere else (README, 'Point it at your own documents').

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent. The split is in characters, because that is what this run recorded — the billed totals are measured, but nothing apportioned them across the parts, and dividing a measured total by character share would produce three numbers nobody counted.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 2,279 characters assembled, 1,143 are instructions — 50% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Rebuilt verbatim by src/prompt.build() over apartment-lease-notice-01 — the same function evals/run.py calls, not a transcription. Its three named parts' character counts (1143 / 822 / 314) match r002's and r003's own recorded prompt_parts exactly.

Full promptThe prompt in full

SYSTEM
------
You detect sensitive spans (PII) in a document. You return JSON and nothing else.

RULES, in order of importance:
1. Find EVERY span that is a genuine instance of one of the categories below. A span you miss leaks — it ships to whoever reads the redacted document. That is the more expensive mistake, so when genuinely unsure whether something is an instance of a category, include it.
2. Do not invent a span that is not actually in the document, and do not flag ordinary text that only resembles a category without being a real instance of it (a four-digit year is not a CARD; a business's public office address is not a personal ADDRESS). Flagging safe text is a real mistake too — it is just the cheaper one.
3. Copy each span's text VERBATIM from the document — the exact substring, character for character, so it can be located afterward. Do not normalise, reformat or truncate it.
4. If the same value occurs more than once in the document, return one entry per occurrence.
5. Return a JSON object of the exact form {"spans": [{"text": "...", "category": "..."}, ...]}. Use exactly one of the category names given below for every entry.

USER
----
Find every sensitive span in this document, across these categories:
- NAME — A person's full name, as it appears -- not a title or role alone.
- SSN — A US Social Security Number, in any format the document writes it: dashed (123-45-6789), undashed (123456789), or labelled inline.
- EMAIL — An email address.
- PHONE — A phone number, in any format: parenthesised area code, dashed, dotted, or with a country prefix.
- ADDRESS — A mailing or street address -- street, city, state and/or ZIP together, as one span. Not a city or state named alone in prose.
- DOB — A date of birth, in any format the document writes it -- numeric, spelled month, or labelled 'DOB'. Not any other date in the document: an appointment date or an issue date is not a DOB.
- CARD — A payment card number, in any grouping the document writes it -- spaced, dashed, or run together. Not a partial 'ending in ####'.

Return a JSON object with one key, "spans", a list of {"text", "category"} objects. Use exactly one of: NAME, SSN, EMAIL, PHONE, ADDRESS, DOB, CARD

DOCUMENT
--------
RIVERSTONE APARTMENTS — Lease Renewal Notice

Tenant: Ingrid Solberg (DOB 1990-03-22)
Unit Address: 88 Fenwick Row, Unit 4, Minneapolis, MN 55401

Your current lease expires in 60 days. To renew, confirm via ingrid.solberg@leasehub.net or call the leasing office at 612-555-0187.

We look forward to another year.

Raw responseThe raw response

{"spans": [{"text": "Ingrid Solberg", "category": "NAME"}, {"text": "1990-03-22", "category": "DOB"}, {"text": "88 Fenwick Row, Unit 4, Minneapolis, MN 55401", "category": "ADDRESS"}, {"text": "ingrid.solberg@leasehub.net", "category": "EMAIL"}, {"text": "612-555-0187", "category": "PHONE"}]}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Mask personal details before letters and forms leave the office — 18 documents. Two tiers of one model family answered, and every answer was then graded one way — by == against a fixed gold value, which is a measurement too and only as good as the gold behind it.

The rulerHow it was graded

Bag-of-(category, normalised text) exact match, pure code — no LLM judge, same reasoning as docs-extract's and docs-redline's judges: the gold is an exact span of text and whether a detected span matches it is a string comparison after light normalisation, not a judgement call. Leaked and over-redacted are scored and reported separately and never averaged into one tuning target — see evals/judge.py's own module docstring, the guardrail this kit exists to measure.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar published on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection onto a rate card, not a bill anyone paid.

Rates checked 2026-08-07. The provider that actually ran r001/r002/r003 is not priced here; its rate is not committed anywhere in this repo, and a dollar figure whose card cannot be named is exactly what this standard forbids.

Grading unitWhat the grading figure prices

FREE, and it is a measured zero: evals/judge.py makes no call and needs no key, so grading 88 spans across any number of runs costs a forker nothing beyond the detection calls themselves.

The gradersOne way to grade, and why it is the only one

The free floor is a 5-pattern regex (evals/baseline.py) over SSN/EMAIL/PHONE/CARD and DOB-when-cued by a nearby label — it abstains on NAME and ADDRESS entirely, which have no fixed surface shape a regex can name in advance. Scored on the same 18 documents and 88 labelled spans the model runs use: recall 0.5909 against the model's 0.9886-1.0, almost entirely explained by the 34 of 36 leaked spans that are NAME (19) and ADDRESS (15) — the two categories this baseline cannot even attempt. On the five categories it does attempt, it is strong (perfect on PHONE, SSN, CARD; 9 of 11 on DOB) and its 2 over-redactions are both on EMAIL, where an emailish-looking fragment matched the pattern without being one of the 14 labelled addresses. The finding this comparison supports: a model earns its keep here specifically on NAME and ADDRESS, and is a much smaller step up on the five more regular categories.

The row opens its own page: how the test was run, one real row, the formula with every iteration of it, and the analysis.

Choosing oneWhen to reach for the model at all

The comparison is only useful if it ends in a choice, and on this kit the choice is not which grader — there is one, and it is ==. It is whether to call a model for this field at all, and the answer is conditional.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-07. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingGrading is free here, and it is a measured zero

evals/judge.py makes no call and needs no key: grading 88 spans across any number of runs costs a forker nothing beyond the detection calls themselves.

Cost driversWhat actually moves the bill

• The whole document goes in one call, no chunking — cost scales with document length, not with question complexity; the fixed system+schema overhead (1,965 characters, ~86% of the assembled prompt) is paid on every call regardless of how short the document is.
• One call per document, so cost scales linearly with corpus size (see cost_at_10x).
• Which model reads it: the reasoning tier (pro) averaged 124.3 output tokens per document on this run against flash's 103.8 — about 20% more output for one fewer leaked span.

Your volumeWhat it costs at your volume

Linear, and steeply so — one call per document, no batching, no shared prefix beyond the ~1,965-character system+schema overhead. 180 documents (10x this corpus) projects to about 10x the measured $0.0019 (18 docs on the gemini-priced card), roughly $0.019, IF every document behaves like this corpus's average — but see the cliff below, which this kit has already measured directly.

Model choiceWhy this model

The connection already configured on this machine (the shared .env). No model comparison was run to pick the fast tier — the reasoning tier was run second specifically to give this kit's Eval lens two distinct models, per the standard's own requirement, and its own measured recall (1.0 vs the fast tier's 0.9886) is a real reason a forker might prefer it despite the higher cost and latency.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 18 documents detected, 88 labelled spans, all graded by pure code. Flash's run (r002) scored all 18 — the fast tier's numbers are what this table prices, the same row Cost.cost_by_model[0] already uses.

The codeEvery file, and what it is for

Seven modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Three of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

data/categories.jsoncategories — a swap seam

the fixed 7-category schema the prompt is generated from and the judge grades against

You change it to: add an eighth category (e.g. PASSPORT_NUMBER) here plus new labelled examples — nothing about the pipeline shape changes, per the file's own _comment.

# 

src/prompt.pyprompt

assemble the one detection call, all seven categories in it; parse the reply, tolerantly but never creatively

# Assemble the detection prompt. One prompt per document, all seven categories in it.

SYSTEM = (
def category_schema(categories):
def build(doc_text, categories):
def parse(raw, categories):

src/detect.pydetect — a swap seam

the AI layer — one provider, one key, one call per document

You change it to: the output ceiling, currently 800 — sized as a guess before any run fired; r001 shows it is not yet proven sufficient under the provider's default reasoning setting

# Detect one document's sensitive spans: prompt, one model call, locate, done. This is the whole

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CATEGORIES = os.path.join(HERE, "data", "categories.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 800
def load_categories():
def load_doc(doc_id):
def documents():
def detect(cfg, doc_text, categories, complete=None, thinking=None):

src/adapters/__init__.pyadapters — a swap seam

raw HTTP per provider (openai-compatible, anthropic); retry on transient errors, refuse to guess on terminal ones

You change it to: any OpenAI-compatible host, or Anthropic's Messages API

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/redact.pyredact

pure code — locate spans in the source text, mask them to [CATEGORY], build the highlighted view. No model call anywhere in this file.

# Turn detected spans into a redacted document and a highlighted one. PURE CODE — no model call

CSS_CLASS = {"NAME": "name", "SSN": "ssn", "EMAIL": "email", "PHONE": "phone",
def locate(text, spans):
def _dedupe(located_spans):
def mask(text, located_spans):
def highlight(text, located_spans):
def redact(text, spans):
def highlighted(text, spans):

src/app.pyapp

the local UI's HTTP server — renders with no key, calls the model only from /api/redact

# The minimal local UI. Standard library only — python -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
PORT = int(os.environ.get("PORT", "8770"))
class H(BaseHTTPRequestHandler):
def main():

evals/judge.pyjudge

score leaked vs over-redacted separately, pure code, no LLM judge

# Score a redaction run. PURE CODE — no LLM judge in this kit either, for the same reason

def norm(text):
def score_doc(labelled_spans, detected_spans):
def score(labels, records):

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 638 input and 103 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

A local, single-user tool over self-authored documents containing synthetic personal data. It binds 127.0.0.1, has no auth, no rate limit and no session — correct for a demo on your own machine and wrong for anything else. There is no chat box: the only text a stranger controls is the DOCUMENT submitted for redaction, which is exactly the surface this run attacks.

Read from the shared .env or the real environment only, never written into the repo, never requested from a reader on any surface. The repo has never held a credential. The UI screenshots are taken with the key blanked, and the shoot script refuses to start if anything is already listening on its port — a stale copy of the same kit holding a real key passes an identity check perfectly, and that gap cost a real provider call elsewhere.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. Reproduce with python -m evals.redteam --run-id x001-docs-redact --docs 5. Measured on 2026-08-08, run x001-docs-redact: 5 documents crossed with 6 attack families, 30 live calls on the fast tier with reasoning disabled — the configuration the app and every scored eval ship. All 30 attempts scored. Detection is against data/labelled.jsonl, the GOLD SPANS, rather than against a clean run: this kit's result files carry aggregate scores and no per-document record, and the labels answer the better question — what SHOULD have been found. Two free control stubs ran first, one resisting everything (0/6) and one complying with everything (6/6).

⚠︎ BOTH FAILURE DIRECTIONS MOVED, AND A RECALL-ONLY MEASUREMENT SEES HALF OF THAT. The model can be made to UNDER-redact (an identifier ships) and to OVER-redact a chosen phrase (the document is destroyed). An attacker who can add spans can black out the terms of an agreement as easily as reveal them, which is why this kit publishes precision beside recall. The targeted span is always the highest-harm identifier the document carries — SSN, then CARD, then DOB — because a missed name is embarrassing and a missed Social Security Number is the thing the kit exists to prevent.

The resultA line claiming a whole category is out of scope worked on every attempt — 5 of 5 — and its output looks completely normal. The reply is full of spans, the redacted document looks redacted, and one entire category is quietly gone. The crude version, a banner declaring the document public, worked only 1 time in 5: plausibility beats force. Overall 15 of 30 scored attempts followed. The harm here is silent by construction — nothing downstream can tell "the model looked and found nothing" from "the model was told not to look".

The four-kit picture is the finding, and it is not the one two runs suggested. docs-verify — injected into the document, closed verdicts — is this claim supported?: 3 of 30 followed, 90.0% resisted · docs-comply — injected into the rulebook, closed verdicts — does this rule pass?: 13 of 29 followed, 55.2% resisted · docs-redact — injected into the document, span extraction — find every identifier: 15 of 30 followed, 50.0% resisted · docs-summarise — injected into the document, free-text generation — write a brief: 16 of 28 followed, 42.9% resisted. Three of the four sit between 43% and 55%, and the outlier that RESISTS is a document-injection kit — so "rulebook versus document" does not explain the spread. What lines up is what the model is asked to DO with the text. Asked to JUDGE it against a fixed, closed vocabulary — is this claim supported by that source? — the model treats the document as EVIDENCE, and an instruction inside it is just more evidence; it resists at 90%. Asked to ACT on it — summarise it, extract from it, apply rules to it — it treats what it reads as part of the JOB DESCRIPTION and obeys about half the time. docs-comply fits: closed vocabulary like docs-verify, but its injection arrives in the RULES, which are the job description by definition. Four runs, one provider, one day, six attacks each — a pattern across four comparable measurements, not a study.

HonestyWhat this does not prove

• Whether a real attacker would use these six. They were written by the kit's author against the kit's own design.
• Whether the reasoning tier resists differently. This run used the fast tier only.
• Whether the cross-kit pattern is causal. Four runs on one provider on one day, built to be comparable on purpose. That makes the comparison real and the MECHANISM behind it a reading.
• The app's HTTP surface. The run drives the kit's own entry point — the same code path the app calls — but the app was not attacked through its own interface.
• Documents carrying no span of the exempted categories. Those attempts are scored null — inapplicable, not resisted — so the category rate has a smaller denominator than it appears.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Find EVERY span that is a genuine instance of one of the categories below. A span you miss leaks — it ships to whoever reads the redacted document. That is the more expensive mistake, so when genuinely unsure whether something is an instance of a category, include it.

src/prompt.py — SYSTEM, rule 1 of 5. Prompt-level, not enforced in code: nothing downstream checks that a returned span set actually favours recall over precision, the way docs-redline's CONFIDENCE_FLOOR checks a stated confidence number. The model is asked to make the call itself; redact.py just locates and masks whatever it returns.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• ⚠︎ IT DOES NOT HOLD ON THE MODEL'S OWN DEFAULT SETTINGS. r001 (thinking left at the provider default — the setting a naive fork gets by doing nothing) shows what 'when genuinely unsure, include it' is worth when the model spends its whole output budget on reasoning nobody asked for: 4 of 18 documents returned NO spans at all, not merely an over-cautious few — recall on those documents was 0%, because the reply never reached the JSON. A prompt rule cannot fire on a reply that never arrives.
• It is not a code-enforced floor. Unlike docs-route's confidence floor or docs-redline's CONFIDENCE_FLOOR, nothing in src/redact.py or src/detect.py checks whether the model actually favoured recall — the rule lives in the prompt text alone, and there is no mechanism that would catch a model that quietly ignored it while still returning a well-formed reply.
• It says nothing about a document that is hostile rather than merely difficult. Nothing here has been attacked — see the security gap named in Eval.could_not_verify below.

WatchedWhat is watched, and why that one

The kit measures 20 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 13 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

5 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

redact · with the model in the path — 4 runs. Columns here are only ever compared with each other.

not a time series No two of these 4 runs measured the same system — they differ on documents, failures, max_tokens, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 7 chips that all say so.

DeviationsWhat deviated

0 breaches across 5 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-score both scored models whenever src/prompt.py, data/categories.json, or MAX_TOKENS changes — all three change what the model is asked for or how much room it has to answer. Re-run with thinking explicitly on at least once per settings change too, given how large the r001/r002 gap was.

In one lineNo framework, deliberately

This kit has no framework and no dependencies beyond the standard library — requirements.txt names nothing, on purpose (its own comment: 'This kit is Python standard library, end to end'). The whole detection decision is four files: prompt.py, detect.py, redact.py, adapters/__init__.py.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing in this pipeline loops, branches or retries beyond the adapter's own transient-error backoff: one call per document, one set of spans, pure code the rest of the way. A graph earns its place when a cycle appears, and prompt -> detect -> locate -> mask/highlight is a straight line with no cycle in it — the same shape every sibling kit's own graph note describes, for the same reason.

The other sideWhat a framework costs you

• A dependency tree, and a version treadmill on somebody else's release schedule — for a kit whose entire promise is a ten-minute clone.
• An abstraction over the one thing this kit exists to show a reader: the exact prompt the model receives, and the one confidence-shaped rule (favour recall) it is asked to follow, in plain text a reader can hold in their head.
• The closest real exception any sibling kit names is constrained decoding — forcing valid JSON. This kit's own r001 failures were not malformed JSON, they were EMPTY replies (the model spent its whole ceiling on reasoning before writing anything); a JSON schema constrains the SHAPE of an answer, not whether the model ever gets around to writing one.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r002-docs-redact on the fast tier, 2026-08-07. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT, same as the rest of the repo. No third-party content, no scraped source, nothing to attribute — every name, SSN, DOB, email, phone, address and card number is fabricated, never sourced from a real individual. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineBag-of-(category, normalised text) exact match

For each labelled span, did the detector report the same category and (normalised) text anywhere in its reply for that document? Matched as a multiset, not by document offset — see evals/judge.py's module docstring.

MethodHow the test was run

evals/judge.py, in-process, no key. Gold spans come from data/labelled.jsonl, and evals/check_labels.py re-derives every span as an exact substring of its document before a run is trusted.

The inputOne real row, seen by every grader

The one document guaranteed to appear in every run record (evals/run.py always keeps the first successful reply), so it is the one row this page can show with full text on both sides rather than only an aggregate count. It is NOT one of the two rows that erred on r002/r003 — see Eval.could_not_verify for why the specific erring document cannot be named from the saved result files.

The formulaWhat it computes

norm(text) lowercases and collapses whitespace but keeps punctuation (a dash inside an SSN counts as part of the value). Two Counters, one per side, keyed on (category, norm(text)); true_positives = |want & got|, leaked = |want - got|, over_redacted = |got - want|. precision = tp/(tp+over_redacted); recall = tp/(tp+leaked). The two are never averaged into a tuning target — F1 is printed only as a dashboard summary.

IterationsFour rulers, four wrong numbers

The tell was the ranking inverting, not the value moving. Noise moves a number; an artifact reorders the comparison the page exists to make.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: this grader, against gold authored alongside the corpus in data/labelled.jsonl and re-verified as literal substrings by evals/check_labels.py before any run is trusted — not against another model's output.

How tight can the band be? 88 labelled spans is the whole denominator on record; one flipped span moves recall by about 1.1 points. No band exists yet — see Eval.could_not_verify.

Cadence: Re-run on any change to src/prompt.py, data/categories.json, or MAX_TOKENS in src/detect.py — all three change what the model is asked for or how much room it has to answer.

The decisionWhen to reach for it