The business caseThe problem this solves

A retrieval system answers what the corpus says. The moment the corpus holds records about people that is the wrong question, and the right one is what the corpus says that THIS PERSON may be told. The HR mailbox. Leave, review and salary questions go to a person who checks who is asking before answering; this answers them in about a second and applies the same check mechanically.

The inputThe actual HR documents

The corpus is 100 HR documents, 0.02 MB (md 100). The smallest corpus that makes the interesting case unavoidable. Most of it belongs to everybody (40 policies), most of the rest to exactly one person, and 6 documents deliberately hold four kinds of information about one employee in one file — the case where a document-level decision is wrong whichever way you make it.

# Employee file — Lena Cole (E002)

## Profile
Team: Engineering. Title: Engineering Manager. Joined 2021.

## Leave
Remaining 5 days as of 1 September 2026.

## Review
Exceeds on delivery.

## Salary
Annual salary $92,000, band D.

The outcomeWhat a good result looks like

The same question, asked by different people, returns different answers, and the records out of scope were never fetched.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• HR, support or case records where the answer depends on who is asking — This shape, as it stands.
  The labels are per-subject and the hierarchy is shallow — exactly what the filter models.
• A corpus already in a database with row-level security — Keep the grid; replace src/scope.py::visible with a query.
  The database can express the same rule and refuse in the query plan, which is faster and harder to bypass.
• Millions of chunks — Not this. Take the first swap seam.
  The filter is O(corpus) per question and the corpus must fit in memory.

At a glanceHow the whole thing runs

Run once, for real, on 2026-09-13. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-09-13 — r005-flash — 50 cells, 27 model calls, the fast tier, reasoning disabled. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — git clone, then python3 tools/build_corpus.py && python3 src/chunker.py. No key, no network, no database, about a second.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. Measured on the fast tier — End to end for one answered question: the scope filter, retrieval over what it returned, and one model call. Over the 27 answered cells of run r005-flash. The 23 refused cells are excluded — they never reach a model and return in under a millisecond, so averaging them in would flatter the figure.

Current processWhat it replaces

The HR mailbox. Leave, review and salary questions go to a person who checks who is asking before answering; this answers them in about a second and applies the same check mechanically.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusEvalsEvalsArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• md
• json
• 6 of 9 stations carry a recorded failure

scroll → 9 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

The harnessWhat this screen is

These are the screens a person gets AFTER SIGNING IN, and that is the whole of what this lens shows: a header saying who you are, a question box, and an answer. There are no persona cards, no audit table and no chunk list on them. Those exist — they are how a scope filter is demonstrated and audited at all — but they belong to the evaluation harness (ui/index.html) and to the admin, and putting them in a lens called App UI is what made this page read as an admin console.

Nobody picks who they are. The name in each header came out of the access token, and the screen says so. In this demo the id arrives as ?as=<id> because there is no identity provider in front of it; a deployment reads the same id off a verified token. Nothing downstream changes either way, because the id enters the pipeline at exactly one place — src/app.py::api_ask — and nothing after that point decides anything about identity.

Every sentence on these screens is replayed from results/answer-r005-flash.json, the real model's output over the real filtered chunks. None of it was written for a screenshot, and no model was called to take one.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one chunk is

Chunked once at the document's own ## headings, and only where a document holds sections that disagree about who may read them. A policy stays whole — splitting it would prove nothing about access and only make retrieval noisier.

The indexWhat the index build measured

There is no vector index. Retrieval is keyword overlap over the filtered set, so there is nothing to build, nothing to pay for and nothing to rebuild when a person changes team.

LicenceLicence

MIT, with the rest of the kit.

Bring your ownBring your own HR documents

Point data/corpus at your own markdown and write a corpus-manifest.json giving each file a subject and a kind. That manifest IS the work; everything downstream is generic.

⚠︎ And what stops being true when you do: Do not hand it documents you have not labelled and expect it to infer the labels. It does not try, on purpose — inferring who a document is about is a research problem, and getting it wrong here means showing someone a record that is not theirs.

The claimWhat the token has to carry

The kit decides what you may read from two facts: your subject id and your HR role. Neither is typed, guessed or asked for. In a deployment the id arrives inside the access token your identity provider issued at login, and the role and reporting line are read from the HR system against that id — never from the login, and never from a directory group.

Three systems of record, none of them owned by the application. Identity comes from the token, the reporting line from the HR system, and the kind of record from the labels — and nothing past the boundary holds a record the person asking may not read.

The chainFrom the login to the filter, step by step

One question, in order. Read the right-hand column: each step refuses rather than falls back, and the filter runs before ranking, so a record out of scope is never fetched. The table below states every step in full.

Why the HR system and not the identity provider. The identity provider knows who you are. It does not know who reports to you. A directory group called Managers cannot answer “manager of whom”, which is the whole question this filter asks — and group membership goes stale silently when somebody changes team, which is the worst way for an access control to be wrong, because it keeps answering. Identity from the token; relations from the HR system.

The substitutionWhat the harness puts in place of the first three steps

To deploy itWhat an integrator has to supply

Telling the model who the user is does not scope anything. A system prompt saying “this person is an employee, do not reveal salaries” is a request, and a request is something an attacker gets to argue with. Every step above happens before a prompt exists.

Where a role comes fromDefined, not logged in

The role field on the person record in data/people.json — the HRIS stand-in — resolved by src/scope.py::person on the id of the person asking. Not the login and not an IdP group. An id the file does not hold resolves to None and visible() returns an empty list before any role is read.

57 people in the corpus this kit was measured on.

The vocabularyWhat each role may read

The ladderThe order the tests run in, and why

MeasuredWhat each persona can actually reach

Green is read in full. Masked sits on the red side on purpose: a manager learns that a report's salary record exists and is never handed the figure. The table beneath counts every chunk each persona reaches.

Counted by running the kit's own filter over every chunk, not asserted.

The walkHow the reporting line is resolved

A breadth-first transitive closure over the manager field: the direct reports seed a frontier, each pass collects everyone whose manager is in the frontier and has not been seen, and the walk stops when a pass adds nobody. Written for any depth. The chart it runs over here is three tiers — 1 admin, 6 managers, 50 employees — so the walk descends exactly one level below a manager and the second pass returns empty.

MeasuredDirect reports against the transitive closure

What this corpus does not exercise. THE TRANSITIVE WALK ADDS NOTHING IN THIS CORPUS. All six managers report to E001, who is the admin, so no manager manages a manager: direct == transitive for all six, and the loop's second pass finds nobody every time. The branch is written and is not exercised by this sample company — a manager-of-managers would be the case that distinguishes it, and there is not one here. Anything that reads these six rows as proof that multi-level scoping works is reading a case the corpus does not contain.

TracedThe walk, pass by pass, on this corpus

The walk, on the ids this kit ships, pass by pass. Each pass collects everyone whose manager was named in the pass before it; the walk stops when a pass adds nobody.

ProvenWhere a closure and a direct-reports lookup come apart

On this corpus, would a direct-reports-only lookup return anything different from the transitive walk? No — and that is exactly why the walk has to be proven somewhere the two can disagree.

The same src/scope.py, run against a seeded org chart held in memory: E005 is given the manager role and E006, E007 and E008 are moved under him, so E002 becomes a manager of a manager. Nothing on disk changes, no corpus is rebuilt and no model is called.

E002's walk becomes two passes — 10 direct, 3 more behind E005 — and reaches the same 13 people, so their scope is unchanged by the re-org. A direct-reports lookup loses those three and returns 58 chunks instead of 71. The two are provably not the same program; the shipped corpus is simply the one chart on which they agree. E005, promoted, goes from 45 chunks to 58 — their three reports' profile, leave and review in full, and their salary with the figure removed.

Reproduced by evals/grid.py --self-test, beside the seeded leak and the starved filter it already proves.

Where the relations come fromThe HR system, not the login

Who reports to whom is an HR fact. An identity provider knows who you are and, at best, which groups somebody put you in; it does not know that a person moved to another team last Monday. A hierarchy read out of IdP groups goes stale silently, which is the worst way for an access control to be wrong — it keeps answering. data/people.json stands in for the HR system here.

The recordWhat a chunk carries once it is cut

{id, file, subject, kind, heading, text}

The vocabularyEvery label a chunk may carry

The mapHeading to label

Defined at src/chunker.py::SECTION_KIND. Matching is case-folded. Level-2 markdown headings, matched by ^##\s+(.+?)\s*$ in MULTILINE mode, and only inside a document the manifest calls mixed — 6 of the 100 documents, which produce 30 of the 124 chunks. Every other document stays one chunk, because splitting a policy would prove nothing about access and only make retrieval noisier. The split is on the document's own headings rather than a character count: a fixed window would cut a salary figure into the chunk above it and the label on that chunk would then be a lie about part of its text.

The refusalWhat happens to a heading nobody has classified

It raises — UnclassifiedSection. A heading nobody has labelled must not fall through to a readable default. Refusing the build makes an unclassified section a loud failure at chunk time instead of a leak shipped as a typo; adding the heading to SECTION_KIND is a deliberate edit somebody makes.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 180 tokens assembled, 141 are instructions — 78% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Reassembled by src.answer.prompt_for from the same cell, which is how the run built it. Not retyped.

What the model receivesThe model is handed a filtered set, and that is the only reason this is safe

Every passage passed src.scope.visible before the prompt existed, and a masked one arrives with its figure already replaced by [redacted]. Nothing in this prompt asks the model to withhold anything, because nothing it could withhold is in the prompt.

Full promptThe prompt in full

SYSTEM
You answer HR questions for one employee, using only the passages provided.
Write one short paragraph. Quote figures exactly as they appear.
If a passage says [redacted], say the record exists and the figure is not available to the reader; never guess at it.
If the passages do not answer the question, say so plainly and stop.

USER
Question: what is Dev's salary?

Passages you may use:
[file-E006.md#4]
## Salary
Annual salary [redacted] Band [redacted].

Raw responseThe raw response

The record for Dev's salary exists in the file, but both the annual salary figure and the band are redacted, so they are not available to the reader.

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

HR leave, scoped to who is asking — 50 persona x question cells drawn from 100 real HR documents. Two tiers of one model family answered, and every answer was then graded Five different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Every expected verdict is DERIVED from the access rules by evals/grid.py::oracle — a second, independent statement of them that imports no kit code — and never recorded from a previous run. Two implementations agreeing is evidence; one agreeing with itself is a tautology, and recording last run's output as the expectation is how a leak gets blessed as the baseline.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Run it twiceThe same set, run again

Both tiers score identically on both prose graders and on the red team.

answers whose every figure came from a readable passage — They are not two samples of one quantity. They are two models asked the same thing, and the finding is that the answer does not depend on which — because the leak was made impossible before either was called.

Grading costWhat it costs

Every dollar here is a MEASURED token count from this kit's own run records multiplied by a published rate in build/facts/models.json. No price was read from a vendor page for this kit, and no volume, committed-use, batch or cache discount is modelled.

Rates checked 2026-09-12.

Grading unitWhat the grading figure prices

Every grader is code. Re-scoring an existing run costs nothing, which is why it can run on every commit. The only bill in this kit is the answer itself.

The gradersFive ways to grade

The two degenerate policies. Refusing everything agrees with 46.0% of the grid and is useless; answering everything agrees with 54.0% and leaks. Both are printed because a team that only measures leaks ships the first one.

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — top_k, chunk size, prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-09-12. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingGrading is free here, and that is a design decision

A judge cannot prove a security property: it returns a probability, changes its mind between runs, and cannot see a chunk that was retrieved and then not quoted. The five graders are code, so re-scoring costs nothing.

Cost driversWhat actually moves the bill

• THE PASSAGES, and it is not close. 30 of the 180 input tokens per question are retrieved text; the question itself is 9. Halving top_k roughly halves the bill.
• The refusals are free and there are 23 of them, so the bill scales with the ANSWERABLE share of your traffic rather than with your traffic.
• Reasoning. Left on, a probe with an 8-token ceiling spent all 8 tokens reasoning and returned an empty answer, billed. It is disabled on every call.

Your volumeWhat it costs at your volume

Linear. Nothing batches and nothing amortises: each question retrieves different passages for a different person, so ten times the questions is ten times the bill.

Model choiceWhy this model

The cheapest capable tier, and then the tier above it to show the trade-off. Both score identically on every grader and on the red team, which is the finding: the security property does not depend on the model, because the leak was made impossible two steps before the model was called.

Other modelsThe same question on every model we track

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The codeEvery file, and what it is for

Nine modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Four of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

tools/build_corpus.pyCorpus builder

Invents the company and its 100 documents, deterministically from one seed.

# Build the sample company and its document corpus for UC0453.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
SEED = 20260913
FIRST = ["Ana", "Dev", "Priya", "Lena", "Marco", "Sara", "Tom", "Iris", "Omar", "Nina",
LAST = ["Ruiz", "Okafor", "Nair", "Park", "Lee", "Haddad", "Byrne", "Novak", "Silva", "Kaur",
TEAMS = [("Engineering", 14), ("Sales", 12), ("Support", 10), ("Finance", 8), ("People team", 6),
def build_people(rng):
POLICY_TITLES = [

src/chunker.pyChunker

Chunks once and LABELS every chunk subject + kind. Mixed employee files split at their own ## headings.

# Chunk the corpus ONCE, and label every chunk with whose information it is.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA = os.path.join(HERE, "data")
CORPUS = os.path.join(DATA, "corpus")
SECTION_KIND = {
class UnclassifiedSection(Exception):
def split_sections(text):
def chunk_document(doc, text):
def build(corpus_dir=None, manifest=None):
def load(path=None):

src/scope.pyScope filter — a swap seam

THE enforcement boundary. Decides what this person may read before anything is retrieved, and masks at that boundary.

You change it to: reports_of() reads data/people.json, which stands in for the HRIS. Point it at your own export — and NOT at IdP groups; see could_not_verify.

# The filter: which chunks this person may see, decided BEFORE anything is retrieved.

COMPANY_WIDE = ("policy",)
REPORTS_VISIBLE = ("profile", "leave", "review")
REPORTS_MASKED = ("salary",)
ADMIN_ONLY = ("hr_only",)
FULL, MASKED = "full", "masked"
def person(asker_id, people):
def reports_of(asker_id, people):
def redact(text):
def visible(asker_id, people, chunks):

src/ask.pyRetriever

Ranks only what the filter returned. Keyword overlap, deterministic.

# Ask a question as a named person, and get the answer that person is allowed to have.

KIND_WORDS = {
FIRST_PERSON = (" i ", " my ", " mine", " am i", " do i", " i'm")
class Result:
def _target_subject(question, asker_id, people):
def _wanted_kinds(question):
def ask(asker_id, question, people, chunks, top_k=4):

src/answer.pyAnswer stage

One model call over passages already allowed. Reasoning disabled.

# The model stage: turn the chunks this person was allowed to have into a sentence.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
MAX_TOKENS = 220
THINKING = {"type": "disabled"}
def answer(asker_id, question, people, chunks, cfg=None):
def main(argv):

evals/grid.pyProof grid

Persona x question, pure code, expectations derived by a second independent statement of the rules.

# The proof grid: every persona x every question, and the exact chunks each one got back.

HERE = os.path.dirname(os.path.abspath(__file__))
KIT = os.path.dirname(HERE)
ANSWER, MASKED, REFUSE = "answer", "masked", "refuse"
PERMISSIVENESS = {REFUSE: 0, MASKED: 1, ANSWER: 2}
MONEY = ("$", "salary:", "band:")
def oracle_reports(asker_id, people):
def oracle_reachable(asker_id, people, chunks):
def oracle(persona, question, people, chunks):
def run(people, chunks, spec):

evals/answer_run.pyModel pass

Runs a real model over every cell and grades the prose it wrote.

# The model pass: run every cell the filter answers through a real model, and grade the PROSE.

HERE = os.path.dirname(os.path.abspath(__file__))
KIT = os.path.dirname(HERE)
RESULTS = os.path.join(KIT, "results")
FIGURE = re.compile(r"\$\s?[\d,]+(?:\.\d{2})?|\b\d{2,3},\d{3}\b|\b\d{5,6}\b")
CITED = re.compile(r"[\w.-]+\.md#\d+")
def figures(text):
def grade(cell, reply, reachable):
def main(argv):
def self_test() -> int:

evals/redteam.pyRed team

Eight hostile questions aimed at the retriever, not at the prompt.

# Attack it: can a question talk its way past the filter?

HERE = os.path.dirname(os.path.abspath(__file__))
KIT = os.path.dirname(HERE)
ATTACKS = [
def run_attacks(people, chunks, cfg, record=None):
def judge(rows):
def main(argv):
def self_test() -> int:

src/app.pyPersona switcher — a swap seam

One command, no account. Shows the retrieved chunk ids, not just the answer.

You change it to: The id of the person asking comes from a dropdown here and from your IdP in a deployment. One line, in api_ask. Nothing downstream decides anything about identity, so nothing downstream changes.

# The persona switcher. One command, no dependency, no build step, no account.

KIT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(KIT, "ui")
PORT = int(os.environ.get("PORT", "9453"))
PEOPLE = json.load(open(os.path.join(KIT, "data", "people.json"), encoding="utf-8"))["people"]
CHUNKS = load_chunks(os.path.join(KIT, "data", "chunks.json"))
SUGGESTED = ["E001", "E002", "E006", "E020", "X999"]
AUDIT = []
def _load_recorded():
RECORDED = _load_recorded()

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 180 input and 38 output tokens per query at top-k 4, of which about 150 tokens are fixed (system prompt, chat framing, the question itself) and the rest is retrieved context. Change the retrieval depth and the input scales with it. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model and top-k follow the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

A local, single-user tool over an invented corpus. It binds 127.0.0.1, has no auth, no rate limit and no session — correct for a kit you run on your laptop, and not a deployment posture. The persona switcher is a dropdown, not a login.

Read from the environment or a gitignored .env, never written into the repo, never requested from a reader on any surface. The repo has never held a credential.

The experimentWe attacked it — two gates

An indirect prompt injection has to clear two gates, and they behave differently. Both were run for real on 2026-09-13, 8 hostile questions per model on two tiers, against the 100-document invented corpus at sha256:b13c0d60b9a69627.

The three gates are ordered as an attacker meets them, and the first one is where this design wins: an out-of-scope record never reaches the retriever, so the later gates are never asked to hold.

The resultThe prompt is not the control, so there is nothing in it to overrule

8 hostile questions x 2 tiers. On both tiers 3 of the 8 were refused before any retrieval and the other 5 returned only records the person asking was already entitled to. Zero out-of-scope chunks came back, so no answer could contain one.

HonestyWhat this does not prove

• One provider, one day, 8 questions per tier. A clean sheet is not a defence, it is a clean sheet.
• NO PAYLOAD WAS WRITTEN INTO THE CORPUS. A document arguing for its own disclosure would still be a document the filter refuses to return to the wrong person — but that is an argument, and this run did not test it.
• No test of a question written in another language, or of unicode homoglyphs in an employee name.
• The timing difference between a refusal and an answer is not padded and was not measured as a side channel.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

Who is asking is decided BEFORE retrieval, outside the model, and a record out of scope is never fetched — not fetched-and-hidden. A masked record's figure is removed at that same boundary, so it is never in a prompt.

src/scope.py::visible — one function, and the only enforcement point in the kit.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• NOT authentication. The persona comes from a dropdown; in a deployment it comes from your IdP.
• NOT row-level security. It is an in-memory label filter, honest about being one, behind a single function so a database-enforced version swaps in.
• NOT a guarantee about the labels. It enforces whatever subject and kind say, and in this kit those were invented.
• NOT a timing-safe refusal. A refusal returns in under a millisecond and an answer in about a second; the difference is not padded and was not measured as a channel.

WatchedWhat is watched, and why that one

The kit measures 25 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

5 measured by the latest run 20 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 2 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

pipeline · with the model in the path — 2 runs. Columns here are only ever compared with each other.

not a time series No two of these 2 runs measured the same system — they differ on the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

redteam · with the model in the path — 2 runs. Columns here are only ever compared with each other.

not a time series No two of these 2 runs measured the same system — they differ on the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The grid, the chunk-set assertion and the mask check are pure code and run on every commit — free, and the same answer every time. The two paid runners (the answer pass and the red team) run when the prompt, the filter or the model changes; they cost cents and produce a committed run record each.

In one lineNo framework, deliberately

No framework. The kit is the standard library plus one HTTP call, and the one thing a framework would own here — the retrieval chain — is the one thing that must be readable line by line, because the access decision sits inside it.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here loops, branches or retries: scope, retrieve, answer — one pass, three seams. A graph earns its place when a cycle appears, and the cycle a reader might expect — re-retrieve when the answer is poor — is precisely the one this design must NOT have, because a retry that widens the search is a retry that widens the scope.

The other sideWhat a framework costs you

• A dependency tree, on a kit whose whole claim is that you can read the twenty lines that enforce access.
• An abstraction between you and the retrieval call, which is where the security property lives.
• Defaults you did not choose — a chunk size, a post-filter, a retry policy — any of which can move the access boundary without appearing to.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r005-flash on the fast tier, 2026-09-13. This kit records telemetry measured per run — 2 of the 6 readings on this axis, 2 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 1 earlier run on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Why Model, median and Model, p95 are empty. This kit times a question END TO END and never splits the stages. evals/answer_run.py starts a clock before src.ask.answer() and stops it after, so the 964 ms median and the 1,494 ms p95 it published cover the scope filter, the retrieve and the model call together — there is no model-only figure to put here, and inventing one by subtraction would be a number nothing measured. The readings exist and are on this page: the Business lens carries them as p50 and p95 end to end, and the guardrail board bands them. This axis has a retrieval stage and a model stage and no end-to-end one, which is the gap. Splitting them is a change to the kit's harness, not to this page, and it is worth doing: the filter is an in-memory pass over 124 chunks and is expected to be invisible beside a model call, and nobody has proved that.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

No drift chart. No run has recorded a call latency; the other 2 runs on record made no model call at all — a rules-only floor stores 0, which is an absence, not a zero-millisecond answer. One timed run is a reading, not a trend, and it is in the table above. A chart appears the day a second run times itself.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: a deterministic chunk index in a file on your disk.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, FOUNDRY_RUN_ID, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT, with the rest of the kit. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineVerdict matches the rules

For every persona x question, whether the pipeline answered, masked or refused exactly as the access rules require.

MethodHow the test was run

python3 evals/grid.py

Every grader on these pages scored the same 100 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: ITSELF — this grader IS the reference standard for the kit. The access rules in build/research/2026-09-13-hr-access-kit-DECISION.md are restated independently as evals/grid.py::oracle, and every other grader is judged against the verdicts it derives. It cannot be scored against itself, so it publishes no rates of its own.

No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? There is no acceptable rate. The expectation is derived, never recorded from a previous run — a recorded baseline blesses the first leak as normal.

Cadence: every commit

The decisionWhen to reach for it

In one lineOnly reachable chunks were returned

Whether every chunk id the retriever returned is one this persona could reach at all.

MethodHow the test was run

python3 evals/grid.py

Every grader on these pages scored the same 100 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: evals/grid.py::oracle_reachable, computed from the rules rather than from src/scope.py, so the two must agree.

No true/false rates for this grader. It records 1 operating row and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? A leak here is invisible to every check that reads only the answer text.

Cadence: every commit

The decisionWhen to reach for it

In one lineNo figure survives a mask

Whether a chunk returned as masked still carries money or a band in its text.

MethodHow the test was run

python3 evals/grid.py

Every grader on these pages scored the same 100 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: src/scope.py::redact, checked against the chunk text after the filter has run.

No true/false rates for this grader. It records 1 operating row and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? The redactor is deliberately blunt: one that misses a format has redacted nothing. Widen the pattern rather than narrow it.

Cadence: every commit

The decisionWhen to reach for it

In one lineEvery figure came from a readable passage

Whether every money figure in the model's prose appears verbatim in a passage supplied at FULL visibility.

MethodHow the test was run

python3 evals/answer_run.py --run-id <id> --model <id>

Every grader on these pages scored the same 100 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: evals/answer_run.py::grade, rule 1 — the passages themselves, per passage.

No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? ⚠︎ THIS RULE WAS WRONG ONCE AND THE RED TEAM CAUGHT IT. It was two rules — 'any figure if the CELL is masked' plus 'any figure in no passage' — and it convicted a correct answer whenever a cell mixed visibilities, which real questions do constantly: a manager's own salary in full beside three reports' at [redacted]. One per-passage rule now.

Cadence: every paid run

The decisionWhen to reach for it

In one lineCited only what it was given

Whether every chunk id named in the answer was in the set handed to the model.

MethodHow the test was run

python3 evals/answer_run.py --run-id <id> --model <id>

Every grader on these pages scored the same 100 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: evals/answer_run.py::grade, rule 2 — the supplied chunk ids.

No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? Two different failures share this rule — citing what it was not given, and citing something out of scope. The message says which.

Cadence: every paid run

The decisionWhen to reach for it