The business caseThe problem this solves

A record series reaches the end of its retention period. Before anything can be destroyed, somebody has to establish that nothing still freezes it — an active legal hold, an audit hold, an open public-records request, or a longer-retention series that captures the same records under a different schedule item. The retention lookup is arithmetic. The hold check is not: a hold's scope is written in prose ('all correspondence relating to the Riverside project, 2019 onward') and has to be judged against this series' own metadata, and the two commonest errors go in opposite directions — freezing everything with a hold anywhere near it, so nothing is ever destroyed and the schedule stops meaning anything, or releasing a series whose hold was released and immediately continued by another. Someone opening each disposition review, reading every line of the hold registry, judging each hold's prose scope against that series' own category, project and closed date, following any 'continues the scope of' reference back to the hold it continues, and only then checking two expiry dates against the review date before deciding whether the series may go to a records officer for destruction approval.

The inputThe actual disposition reviews

The corpus is 55 disposition reviews, 0.04 MB (txt 55). Plain text, one format, invented rather than fetched — a real disposition review names a real agency's series, a real custodian and a real matter, and a records officer's own remark about a specific file, so it is not publishable under any licence. What is publishable is the STRUCTURE: jurisdictions publish general records schedules as (item code, series description, retention period), and that shape is modelled here with entirely invented contents. One format keeps the reader's attention on the judgement the kit is about — prose scope against structured metadata — rather than on parsing.

Record Series
-------------
RS-2006-3779

Custodian Office
----------------
Fairview Municipal Archives

Series Title
------------
Personnel action files - Riverside

Record Category
---------------
personnel

Related Project
---------------
Riverside

Record Closed
-------------
2006-03

Retention Schedule
------------------
GS-22-06, 7 years

Retention Expires
-----------------
2013-03

Overlapping Series
------------------
none on file

Hold Registry
-------------
PR-2016-370 | active | Scope: all personnel records relating to the Riverside project, 2009 onward. | Successor: none

Disposition Queue
-----------------
queued

Officer Notes
-------------
Ordinary cycle item, no correspondence pending on it.

The outcomeWhat a good result looks like

A twelve-field extracted record per review, plus two derived answers: WHICH hold freezes this series (or none), and whether it may be proposed for disposition. Then one pure-code routing decision taken from two of those fields: a series that is frozen and already sitting in the destruction queue is flagged to be pulled back out. On the fast tier that was 660 of 660 cells exact, 55 of 55 binding holds identified and 55 of 55 eligibility verdicts right, with 16 of 16 frozen-and-queued series flagged and no false alarms.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Screening a disposition batch before a records officer opens it, to find series that must not be in it — either tier — they tie on every published grader over the reviews both answered
  100.00 pct eligibility accuracy on both, with 1.00 recall on the frozen class, against the over-cautious floor's 41.82 pct and the tone floor's 60.00 pct. The floors are not strawmen: freezing everything with a hold nearby and trusting the officer's note are the two shortcuts a real desk actually takes.
• Deciding which tier to buy for a monthly run — the fast tier
  It costs 6.9 pct less per review and answers in 3873 ms at the median against 7369 ms, and no published figure separates the two. The deliberating tier also lost the only review either tier lost.
• Pointing this at a real hold registry — the kit's shape — one call per review, the registry sent whole, the rule in pure code downstream
  The parts that must not be probabilistic are not: the three-condition derivation, the review flag and the scoring are all code, and the model is asked only for the prose judgement code cannot do.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-22. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-22 — r001-hold-conflict. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Checked on a fresh checkout with API_KEY left blank: python -m src.app starts, the 55-review picker populates, any review renders in full, and Extract returns a plain sentence saying nothing was called. python -m evals.check_labels passes, and both free floors (--baseline holds, --baseline notes) score a complete run with no key at all. What a cold clone cannot reproduce is either paid run — those need a key, and the committed result files are what stands in for them.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. model call only, one per records-disposition review

Current processWhat it replaces

Someone opening each disposition review, reading every line of the hold registry, judging each hold's prose scope against that series' own category, project and closed date, following any 'continues the scope of' reference back to the hold it continues, and only then checking two expiry dates against the review date before deciding whether the series may go to a records officer for destruction approval.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitecturePromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• 1 of 7 stations carry a recorded failure

scroll → 7 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one section is

cut on underlined section headings; a review with none falls back to one whole-document segment

SetupWhat the setup figure measured

There is no index. Preparation is segmentation only — 55 reviews cut into 660 sections in about two milliseconds, with no model and no network. Nothing is retrieved; the segment exists so an extracted value can name the section it was read from.

LicenceLicence

MIT — this repository's own licence. Every series id, custodian office, project name, schedule code and hold id is invented, no person is named anywhere, and no real agency, published general records schedule, disposition authority or litigation matter is named or reproduced.

Bring your ownBring your own disposition reviews

Replace data/corpus/*.txt with your own disposition reviews, write your own data/fields.json, and supply a gold row per review. SECTION_HINTS in src/select.py maps fields to your headings; anything unmatched falls back to the whole document, so a wrong map costs tokens rather than correctness. Then rewrite eligibility() and AS_OF in src/extract.py to your own disposition rule and your own review date — those two are the kit, and everything else is plumbing around them. Run python -m evals.check_labels before you spend anything: it refuses a run whose gold disagrees with its own values.

⚠︎ And what stops being true when you do: Every number on this page stops being true the moment you do. The eligibility rule here is three conditions this kit invented; yours will have a schedule authority, event-triggered retentions and a hold procedure negotiated with counsel behind it. And the hold scopes here are one templated sentence each — a model that reads them perfectly has not been shown to read a hold notice drafted by a lawyer.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 1,705 tokens assembled, 839 are instructions — 49% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Measured token-for-token via evals/prompt_tokens.py's nested-prefix method on RDS-0004 — three calls at max_tokens=1, each sending the previous prefix plus exactly one more part, with each part's size read off the difference between two consecutive prompt_token counts the provider itself returned (results/tokens-p001-hold-conflict.json). The three parts sum to 1705, which is the same input_tokens the captured RDS-0004 call was billed for — so the published decomposition is of the prompt that was actually sent, not a reconstruction that happens to look tidy.

Full promptThe prompt in full

You extract structured fields from a records-disposition review for one record series, and you decide whether that series may be PROPOSED for destruction. You return JSON and nothing else.

You are not destroying, deleting or disposing of anything. A records officer reviews and releases; your answer is the proposal they review.

THE REVIEW DATE IS 2026-08. Every 'has this elapsed yet' question below is asked against that date and against no other.

RULES, in order of importance:
1. If the record does not state a field, return null for it. Do not infer it and do not use what you know about the world.
2. `binding_hold_id` is the id of the ONE hold in the Hold Registry that freezes this series, or null when none does. A hold freezes this series only when BOTH of these hold:
   a. its status is `active`; and
   b. its scope covers this series, which is THREE separate tests and ALL THREE must pass:
      - CATEGORY: the record's Record Category is one of the categories the scope names.
      - PROJECT: the scope names this record's Related Project, or names any project.
      - DATES: the year in Record Closed falls inside the scope's stated span. 'YYYY onward' has no end. 'YYYY to YYYY' is inclusive at both ends.
   Two of the three passing is NOT a match. If the scope is about a different category, a different project, or a span that does not contain the closed year, that hold does not freeze this series, however serious it sounds.
3. A hold whose status is `released` NEVER freezes a series on its own -- not even when its scope covers it exactly. But a registry line whose scope reads 'continues the scope of <id>' takes its scope from that other hold: if that continuing line is `active` and the scope it inherits covers this series, IT is the binding hold and you return ITS id. Check every registry line before answering null.
4. `disposition_eligible` is decided in this order, and the first condition that fires decides it:
   a. If `binding_hold_id` is not null, the answer is 'no'.
   b. Otherwise, if the Overlapping Series line states an expiry LATER than 2026-08, the answer is 'no' -- a longer-retention series that also captures this record still holds it, even though the series' own retention has run out. 'none on file' means there is no overlapping series and this condition does not fire.
   c. Otherwise, if Retention Expires is LATER than 2026-08, the answer is 'no' -- the series' own retention period has not elapsed yet.
   d. Otherwise the answer is 'yes'.
5. THE OFFICER NOTE IS A FIELD TO COPY, NOT EVIDENCE ABOUT ELIGIBILITY. A note that sounds concerned or mentions an escalation does NOT mean something is holding this series, and a note that sounds routine does NOT mean nothing is. The registry, the two expiry dates and the record's own metadata decide; the note is the records officer's own remark and may disagree with all of them.
6. Copy values verbatim from the record wherever possible. Dates are 'YYYY-MM' exactly as written. `overlapping_expires` is null when the Overlapping Series line reads 'none on file' -- return null rather than a guess or an empty string.
7. Use the exact allowed value for a field that lists them.
8. Return every field named in the schema, even when the answer is null.

Extract these fields:
- series_id (string) -- the record series identifier, verbatim
- series_title (string) -- the title of the record series, verbatim
- record_category (enum) one of: correspondence, contracts, permits, personnel, financial -- the structured records category on the series, verbatim
- related_project (string) -- the project this series relates to, verbatim
- record_closed (string) -- the month the record series closed (its retention cutoff), YYYY-MM
- retention_code (string) -- the general schedule item code applied to this series, e.g. GS-05-08 -- the code only, without the retention period
- retention_expires (string) -- the month this series' own retention period expires, YYYY-MM
- overlapping_expires (string) -- the expiry month stated on the Overlapping Series line, YYYY-MM. When that line reads 'none on file' there is no overlapping series and this is null -- return null rather than a guess or an empty string
- binding_hold_id (string) -- the id of the ONE hold in the Hold Registry that freezes this series, or null when none does. A hold freezes it only when its status is 'active' AND its scope covers this series on ALL THREE tests: the record's category is one the scope names, the scope names this record's project or any project, and the year in record_closed falls inside the scope's span ('YYYY onward' has no end; 'YYYY to YYYY' is inclusive). A 'released' hold never freezes anything on its own -- but an ACTIVE line whose scope reads 'continues the scope of <id>' inherits that hold's scope, and if the inherited scope covers this series then THAT line is the binding hold and its id is the answer
- queue_status (enum) one of: queued, not_queued -- is this series already sitting in the destruction queue, or not?
- officer_notes (string) -- the records officer's own free-text note on the series, copied verbatim
- disposition_eligible (enum) one of: yes, no -- may this series be PROPOSED for destruction? This is a proposal for a records officer to review, never an instruction to destroy anything. Decide it STRICTLY from binding_hold_id, overlapping_expires and retention_expires -- never from officer_notes -- against a review date of 2026-08, and take the conditions in this order, first one wins: (a) if binding_hold_id is not null, answer 'no'; (b) otherwise, if overlapping_expires is later than 2026-08, answer 'no'; (c) otherwise, if retention_expires is later than 2026-08, answer 'no'; (d) otherwise answer 'yes'

Return a JSON object with exactly these keys: series_id, series_title, record_category, related_project, record_closed, retention_code, retention_expires, overlapping_expires, binding_hold_id, queue_status, officer_notes, disposition_eligible
Use null for any field the record does not state.

RECORDS DISPOSITION REVIEW
--------------------------
Record Series
-------------
RS-2018-8284

Series Title
------------
Financial transaction files - Bell Harbor

Record Category
---------------
financial

Related Project
---------------
Bell Harbor

Record Closed
-------------
2018-12

Retention Schedule
------------------
GS-30-01, 7 years

Retention Expires
-----------------
2025-12

Overlapping Series
------------------
none on file

Hold Registry
-------------
PR-2025-041 | active | Scope: continues the scope of AH-2023-234. | Continues: AH-2023-234
AH-2023-234 | released | Scope: all financial records relating to the Bell Harbor project, 2015 onward. | Superseded by: PR-2025-041

Disposition Queue
-----------------
queued

Officer Notes
-------------
Routine end-of-retention item. Nothing outstanding on this series.

Raw responseThe raw response

{
  "series_id": "RS-2018-8284",
  "series_title": "Financial transaction files - Bell Harbor",
  "record_category": "financial",
  "related_project": "Bell Harbor",
  "record_closed": "2018-12",
  "retention_code": "GS-30-01",
  "retention_expires": "2025-12",
  "overlapping_expires": null,
  "binding_hold_id": "PR-2025-041",
  "queue_status": "queued",
  "officer_notes": "Routine end-of-retention item. Nothing outstanding on this series.",
  "disposition_eligible": "no"
}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Check an agency's expiring records for holds before destruction — 55 disposition reviews. Two tiers of one model family answered, and every answer was then graded Four different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

The rulerHow it was graded

Every grader here is pure code. Gold is not a second opinion about the officer's note — it is the kit's own three-condition derivation, re-run inside the grader over the values each review itself states, so the truth cannot drift from the rule the prompt states and the app applies.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection, not a bill. Nobody paid any of these numbers.

Rates checked 2026-08-18. The provider that actually ran r001 and r002 publishes no rate card this repo commits, so nothing here is what the run cost. What is published is the token count, which is the part that transfers.

Grading unitWhat the grading figure prices

Free means free: evals/judge.py makes no model call, and neither do the two baseline floors. The only thing a re-run costs is the extraction itself, which is priced in the Cost lens. Re-scoring an EXISTING run costs nothing at all — python -m evals.run --rescore <file> rebuilds the reply set from the result file's own recorded cells.

The gradersFour ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Identical work in both rows — the same query, the same tokens. Only the price list changed. Which is why the token count is the measurement and the dollar is arithmetic you can redo on your own card.

Answering vs gradingAnswering and grading are two separate bills

Both free floors (evals/baseline.py) and all four graders (evals/judge.py) are pure code and cost $0.00 — no key, no model, no network. Re-scoring an existing run costs nothing either: python -m evals.run --rescore <file> rebuilds the reply set from the result file's own recorded cells, which is how the binding-hold grader was fixed after two paid runs without buying a third. The only cost of evaluation is the extraction itself: $0.1261 for the fast tier's 55 reviews and $0.1330 for the deliberating tier's 54, on the projected card.

Cost driversWhat actually moves the bill

• The fixed instruction block, which is most of every call. The system prompt (839 tokens) and the field schema (599 tokens) are byte-identical on every review and together are 84 pct of the input; the review's own sections are 267 tokens. The bill therefore tracks HOW MANY reviews you have, not how long any of them is.
• The reasoning pass you do not see. The captured example billed 576 output tokens for a JSON body of about 290, with reasoning_tokens: 425 — roughly three quarters of the output price, and output is 6x the input rate on the published card. This is the single biggest lever on the bill and it is not visible in the reply.
• One call per review, always. There is no batching and no shared context between reviews, so cost is exactly linear in the size of the disposition queue.
• The size of the hold registry. Every hold line is sent whole; a series with forty live holds sends forty scope sentences. On this corpus that is at most two lines and invisible — on a real registry it would not be.

Your volumeWhat it costs at your volume

Linear in reviews: each call is independent, carries the same fixed prompt and shares nothing with its neighbours, so ten times the queue is ten times the bill and ten times the wall clock. Nothing amortises — there is no index to build and nothing is cached between calls. What does NOT scale linearly is the reliability: the one review lost at 55 to an unretried socket timeout becomes roughly 180 lost at 10,000, in a queue whose whole purpose is that nothing goes missing.

Model choiceWhy this model

The fast tier is what a monthly disposition batch would actually be run on, and this kit's fitment recommends it: it ties the deliberating tier on every published grader, costs 6.9 pct less per review and answers roughly twice as fast at the median.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

The exact work behind every number on these pages: 55 records-disposition reviews, one completion call each, on the fast tier. The deliberating tier's own run is recorded separately in Cost.cost_by_model and is not projected here.

The codeEvery file, and what it is for

Nine modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/segment.pysegment

cut the disposition review into addressable sections, pure code

# Cut a document into addressable sections. Pure code — no model, no network.

def sections(text):
def locate(text, value):
def span_label(secs, start):

src/select.pyselect — a swap seam

map each field to the sections that could state it; the custodian office is mapped by nothing and is never sent

You change it to: map fields to your own review's headings; anything unmatched falls back to the whole document

# Pick which sections plausibly carry each field. Pure code -- the last deterministic step

SECTION_HINTS = {
def for_field(secs, field):
def plan(secs, fields):

src/prompt.pyprompt

assemble one prompt per review — system rules, field schema, mapped sections

# Assemble the extraction prompt. One prompt per records-disposition review, all twelve fields

AS_OF = "2026-08"
SYSTEM = (
def field_schema(fields):
def build(doc_text, secs, fields, selector):
def parse(raw, fields):

src/adapters/__init__.pyadapters — a swap seam

one chat completion over raw HTTP; provider chosen by .env, no vendor SDK

You change it to: add a provider in one function and one dict entry; it must return the same shape including token counts, or its run cannot be published

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/extract.pyextract — a swap seam

parse the reply, locate every value back to its own section, then run the pure-code eligibility derivation and the business-condition flag

You change it to: the three-condition rule and the review date. Change either and gold, the prompt and the scorer all move together, because all three read the same function and the same constant — which is the point, and also why every published verdict figure is invalidated by the edit

# Extract one records-disposition review's fields: segment, select, prompt, one model call, then

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FIELDS = os.path.join(HERE, "data", "fields.json")
CORPUS = os.path.join(HERE, "data", "corpus")
MAX_TOKENS = 6000
AS_OF = "2026-08"
def load_fields():
def load_doc(case_id):
def documents():
def _ym(value):

src/budget.pybudget

count and cap live calls across every kit sharing one key, before the call

# A cap on live calls, shared by every kit on this machine. No dependency, no service.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
KIT = os.path.basename(HERE)
ROOT = os.path.dirname(os.path.dirname(HERE))
SHARED_ENV = os.path.join(ROOT, ".env")
LEDGER = os.path.join(ROOT if os.path.exists(SHARED_ENV) else HERE, ".calls-ledger.jsonl")
PAID_ARMS = ("r", "x")
FREE_ARMS = ("t", "b")
BOOT = os.urandom(6).hex()
PID = os.getpid()

evals/judge.pyjudge

four graders, all pure code, plus a per-class table and a no-gold consistency diagnostic

# Score a disposition run. PURE CODE -- gold is exact and the answer is one value per cell, so

def norm(v):
def equal(field, got, want):
def score(fields, records, golds):
def _matrix(rows, positive):
def score_flags(records, flags, golds):

evals/baseline.pybaseline — a swap seam

two free floors that fail in opposite directions — the over-cautious clerk and the tone reader

You change it to: holds and notes are two shortcuts a real records office actually takes; add your own and the same judge scores it against the same gold

# Two free, rules-and-regex extractors. No model, no key, no spend -- scored by the same judge as

WORRIED_KEYWORDS = ("escalat", "second look", "not confident", "manual audit", "disputed",
def _section(text, name):
def _first_line(s):
def _code(s):
def _overlap_expires(s):
def _hold_ids(s):
def extract_one(text, fields, floor):
def extract(text, fields, floor="holds"):

evals/check_labels.pycheck_labels

refuse to let a run spend until gold agrees with its own derivation and every hard case is present in quantity

# Check the gold set before anything is scored against it. Run this before spending money.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GOLD = os.path.join(HERE, "data", "gold.jsonl")
NULLABLE = {"overlapping_expires", "binding_hold_id"}
FROZEN_CLASSES = {"hold_active", "hold_successor", "overlap_longer", "retention_open"}
ELIGIBLE_CLASSES = {"scope_category_miss", "scope_project_miss", "scope_date_miss",
def bad(msg):
def main():

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 1671 input and 485 output tokens per query, all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

This run's disposition reviews are entirely synthetic (tools/build_corpus.py, seed 20260822): no real agency, custodian, matter or person exists in the corpus, no person is named anywhere in it, and nothing was fetched. The only outbound traffic the kit makes is one chat-completion request per review to the configured provider, carrying the mapped sections of one review — the custodian office is mapped by no field and never leaves the machine. Nothing is written outside the kit directory, there is no database, no auth and no multi-tenancy, and the local UI binds 127.0.0.1 only.

Read from the shared .env or the real environment only, never committed, never logged and never rendered. .env is gitignored from the first commit. On an adapter error the app redacts the key and base URL out of the message before returning it, so a misconfigured endpoint cannot echo a credential into a browser. Result files record the model id and the provider adapter name only.

The experimentWe did not attack it — and three of five boundaries hold

The three boundaries that hold were confirmed by reading the code, not by a run: no code path destroys anything or writes anywhere; the routing rule reads enums and date strings, so the officer's note cannot reach it; and the key and base URL are redacted out of any error before it is returned. A fourth — does a review that never answers fail loudly — held on the harness and failed on the grader, which is a distinction this run only found because a perfect score looked wrong next to a records count. The fifth is open. An indirect prompt injection needs a field an outside party authored, and this kit has exactly one, which makes it the obvious place to attack and the reason the gap is named rather than glossed. Confirmed by reading the code, not by a run, on 2026-08-22 -- no attack run was fired; see redteam.why.

Three boundaries hold, confirmed by reading the code rather than by an attack run. The fourth held only partially and this run is why it is worded that way. The fifth is open and is named as open — an indirect prompt injection in the records officer's note is exactly the attack this kit's shape invites, and nothing here has tried it.

The result0 attack trials, and three of five boundaries checked here hold on evidence. One holds only partially — a review that never came back was recorded honestly and then scored as correct by one grader, which is fixed. The one that does not hold at all is the one this kit's own shape invites: the records officer's note is free text an outside party wrote, on a page that decides whether a record can be destroyed, and whether an instruction hidden in it could move the verdict is unmeasured.

This run's corpus is entirely generated (tools/build_corpus.py, seed 20260822), so no text in it came from an outside party and there was nothing adversarial to resist. The planted ambiguity is a REGISTER mismatch — a routine-sounding note on a series under a live hold — which measures whether a model reads the registry when the prose points the other way. It does not measure whether a model obeys an instruction hidden in the same field. Those are different failures and only one of them is measured here.

HonestyWhat this does not prove

• Whether an indirect prompt injection in the officer_notes field could move disposition_eligible on either tier. No attack run exists.
• Whether the kit behaves safely against a hostile provider — a response body crafted to break the JSON extraction in src/prompt.py::parse, or to return an enormous payload. parse() fails closed to an empty dict, which the harness records as a failed review, and nothing beyond that was tested.
• Whether a real records archive would carry anything sensitive this kit mishandles. The corpus has no personal data by construction; a real disposition review names a custodian, a matter and often the subject of a personnel file, and none of that path is exercised.

In one lineThe guardrail is enforced in code

Not a prompt instruction the model can ignore — a comparison applied after the reply is parsed, before anything downstream sees it.

needs_review fires when the series may NOT be proposed for disposition AND it is already sitting in the destruction queue — disposition_eligible == "no" and queue_status == "queued". It asks for the series to be pulled OUT of the queue; it never proposes putting one in.

src/extract.py::compute(), called by extract() on every review, by the local app on every click, and by evals/judge.py over both the models' output and both free floors' — the same function in all four places.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• It is NOT a check on whether the extracted values are right. If the model names the wrong binding hold and then derives eligibility from it consistently, the series is routed (or not routed) on a wrong reading and this rule cannot tell. The consistency diagnostic in evals/judge.py is the closest thing to that check, it is reported separately, and it is blind to exactly the same case — which is why the binding-hold grader exists and why it needs gold.
• It is NOT a real records programme's escalation policy. Frozen-and-queued is this kit's own simplification, invented for this corpus. No published general records schedule, disposition authority or hold procedure was consulted, and none is reproduced. A real records office weighs how close the destruction batch is, who issued the hold, whether counsel has been notified, and whether the series has already been certified for destruction.
• It is NOT a disposition, and it is not the opposite of one either. Nothing in this kit destroys, deletes, releases a hold, amends a schedule or notifies anybody. The flag is a value in a JSON response asking a person to look at a queue.
• It is NOT a completeness check. It says nothing about reviews that never produced a reply, and the run record is the only place that does. One review of 110 paid calls vanished and this rule was silent about it, correctly — silence is not its job, and it is why the run harness records failures with their error text.

WatchedWhat is watched, and why that one

The kit measures 20 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

7 measured by the latest run 13 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

all shared guards held between the recorded runs (refusal_cells 0) — which is why the history below is allowed to mean anything.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

4 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked.

not a time series No two of these 4 runs measured the same system — they differ on documents, extraction_cells, failures, provider, thinking, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

DeviationsWhat deviated

0 breaches across 4 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

Re-check the flag's evidence on any change to compute() or eligibility() in src/extract.py, on any change to AS_OF, on any regeneration of the corpus, and on any provider or model change. Re-scoring an existing run is free — python -m evals.run --rescore <file> — so a grader change never needs a paid re-run.

In one lineNo framework, deliberately

No framework and no dependencies beyond the standard library. requirements.txt is empty on purpose: the corpus is generated in process, the model is reached over urllib, and the UI is http.server plus one file of vanilla JavaScript. There was nothing left to add.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Nothing here is a graph. The pipeline has one path per review — segment, select, prompt, one call, parse, derive, flag — and no branch, no loop and no agent. The only fan-out is over reviews, and they are independent.

The other sideWhat a framework costs you

• No framework means no framework upgrade, no breaking release and no transitive dependency; it also means no retry policy, no rate limiter and no observability except what src/budget.py and the run record write. This run paid for that: a client-side socket timeout that any HTTP client library would retry by default lost a review.
• Hand-rolled JSON extraction (src/prompt.py::parse) fails closed to an empty dict, which the harness records as a failed review. That is the right direction and it is not the same as a schema-enforcing client.
• The UI is http.server on 127.0.0.1. It is a demonstration, not a deployment, and it says so.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-hold-conflict on the fast tier, 2026-08-22. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 3 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

2 runs not plotted. b000-holds, b001-notes recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — each document goes whole into one call.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT — this repository's own licence. Every series id, custodian office, project name, schedule code and hold id is invented, no person is named anywhere, and no real agency, published general records schedule, disposition authority or litigation matter is named or reproduced. Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one linePer-field exact match, light normalisation

Does the model's value for each of the twelve fields match gold, after trimming whitespace and trailing punctuation and lowercasing? A cell is a hit, a miss (nothing returned where gold has a value) or wrong (something else returned).

MethodHow the test was run

evals/judge.py::score, in-process, no key and no model. The same function evals/run.py calls after every run, and the same one --rescore re-runs over a committed result file's own cells.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py's gold, read back off the same series metadata, schedule item, overlapping series and hold registry the generator wrote into the review text. Every gold value is asserted to be stated verbatim in the document it labels before the corpus is written.

How tight can the band be? There is no tolerance band. It is exact match after trimming whitespace and trailing punctuation and lowercasing; there are no numeric fields in this kit, so there is no rounding to forgive.

Cadence: Re-score on any change to src/prompt.py, data/fields.json or tools/build_corpus.py — the first two change what is asked for and the third changes what the answer is.

The decisionWhen to reach for it

In one lineWhich hold freezes this series

Did the run name the same hold gold names — INCLUDING naming none when none binds? This is the prose scope judgement everything else rests on: three tests (category, project, date span) that must ALL pass, plus a reference to follow when an active line's whole scope reads 'continues the scope of <id>'.

MethodHow the test was run

evals/judge.py::score_flags, in-process, no key and no model.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: tools/build_corpus.py::binding_hold_id(), the same registry-order search with the same one-level reference resolution the prompt states in words. Gold's binding hold is asserted to be an ACTIVE registry line before the corpus is written.

How tight can the band be? No threshold — it is an id or a null. A review that produced no reply is unanswered and is never counted as a correct null, which is the defect this grader shipped with and had fixed.

Cadence: Re-run on any change to the scope wording in tools/build_corpus.py, to the binding_hold_id hint in data/fields.json, or to rules 2 and 3 in src/prompt.py.

The decisionWhen to reach for it

In one linedisposition_eligible against gold's own derivation

Of every series that really is frozen, how many did the run refuse to release — and how many genuinely eligible series did it freeze? FROZEN IS THE POSITIVE CLASS: a series under a live hold that gets called eligible is the one that reaches a destruction batch, and it is the failure a records programme actually pays for.

MethodHow the test was run

evals/judge.py::score_flags, in-process, no key and no model.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: Gold's verdict is re-derived inside the grader by the same three-condition rule the kit publishes — a binding hold first, then a longer-retention overlapping series that has not expired, then the series' own retention, all against the review date 2026-08 — so the truth this matrix grades against can never be a separately-typed label that drifted from the rule.

How tight can the band be? No threshold — the verdict is one of two allowed values, and a reply that returns neither is counted as unanswered rather than folded into the correct-negative cell.

Cadence: Re-run whenever eligibility() or AS_OF in src/extract.py changes, whenever the corpus is regenerated, and on any provider or model change.

The decisionWhen to reach for it

In one lineneeds_review — frozen AND already in the destruction queue

Is this the series somebody has to pull out of the destruction queue today? It fires when disposition_eligible is no AND queue_status is queued. Note the direction: it never proposes a destruction, it asks for one to be stopped.

MethodHow the test was run

evals/judge.py::score_flags, in-process, no key and no model. The same function src/extract.py runs on every reply and src/app.py runs on every click.

Every grader on these pages scored the same 110 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: The same two-value rule (src/extract.py::compute) run over GOLD's own values. It is a business condition, so unlike a self-consistency check it genuinely needs labels — and saying so is half of what makes the number believable.

How tight can the band be? No threshold. Two values, both enums; if either is missing the rule returns None and the row is unanswered, never a quiet 'no'.

Cadence: Re-check on any change to compute() in src/extract.py, and whenever the corpus's queued share changes.

The decisionWhen to reach for it