The business caseThe problem this solves

A hold or retention rule is written against record CLASSES and has to land on SYSTEMS, and systems are where it goes wrong. The register shows the hold applied; one system took the flag and dropped it. Worse, reach accumulates and decays across cycles — a system onboarded two cycles ago is in scope now and its first cycle never mentioned it; one decommissioned last cycle still shows in the register. A reviewer working one cycle at a time cannot see either. an e-discovery register whose APPLIED column is a checkbox — which cannot show a system the connector silently refused, re-raises rules answered two cycles ago, and still lists systems decommissioned last quarter.

The inputThe actual rule review extracts

The corpus is 120 rule review extracts, 0.87 MB (txt 120). A matter register names live litigation, its custodians and every system a company runs. There is no public one and there will not be. A scrubbed export is worse rather than better: scrubbing removes the reviewer's typed note about why a connector refused, which is exactly the layer this kit measures.

Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated records-retention review extract for an
AI use-case kit; it reproduces no company, no matter, no system, no real person and no
real retention schedule. The retention periods and the disposition grace band are
ILLUSTRATIVE, OPERATOR-TUNABLE DEFAULTS -- not any company's or any jurisdiction's
schedule.

Matter Or Rule
----------------------------------------------------------------
Rule reference                     : RR-0001
Rule kind                          : SCHEDULE
Matter                             : -- none; this is a scheduled retention rule
Record class mapping on file       : Supplier correspondence
Records owner (register of record) : Declan Ashworth
Register status                    : ISSUED 2025-05-12, open

Systems In Scope
----------------------------------------------------------------
The matter register's OWN list of systems against this rule, with its OWN
APPLIED / NOT APPLIED column exactly as it stands.

⚠︎ THIS TABLE IS STALE IN BOTH DIRECTIONS. It does not list a system onboarded in an
earlier review cycle, and it still lists one decommissioned in an earlier cycle. And the
column records what somebody DID, not what LANDED: it reads APPLIED for a system that
rejected the flag.

  System                         Kind                       Register says
  M365-MAILBOX-SUPPLY            mailbox                    APPLIED
  DOCSTORE-SHAREPOINT            document store             APPLIED
  SFTP-SUPPLIER-DROP             file drop                  APPLIED

Retention Policy (Default)
----------------------------------------------------------------

The outcomeWhat a good result looks like

every rule's verdict with the unreached system named and its record class — 10 false actions across 94 quiet readings.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Every connector failure is written back into the register as a status — the free floor — structured-log-strong, $0.00
  It BEATS the model on the discriminator (90.0 against 87.5), on raise-it-now, on false actions and on disposition recall.
• Connector refusals get typed into a reviewer note and the register catches up later — the model — but for ONE column, and know which
  Unreachable systems recorded only in prose: 100.0 pct against 47.06 pct. Everything else it either ties or loses.

And where nothing here is good enough:

• You want the reading but cannot carry state between cycles — neither, as configured here
  The stateless control collapses to 53.33 pct on verdict with 62 false actions.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-25. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-25 — r001-retention-reach. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Every corpus extract, the answer key, all three free floors, the injection probe and what r001-retention-reach actually answered ship in the repo. python3 -m evals.check_labels and python3 -m src.app both run with no key and no network.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one reading — one rule, at one scheduled run, end to end including provider-side reasoning tokens, on the shared connection. Not a cold-start figure and not a per-rule SLA: 120 readings ran in 557.6 wall seconds because different rules run concurrently while a single rule's runs are strictly serial.

Current processWhat it replaces

an e-discovery register whose APPLIED column is a checkbox — which cannot show a system the connector silently refused, re-raises rules answered two cycles ago, and still lists systems decommissioned last quarter.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• 3 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one reading is

40 rules x 3 review cycle, walked strictly in order per rule so the carried state moves forward the way it would in a real deployment. Different rules run concurrently; a single rule's cycles never do.

SetupWhat the setup figure measured

There is no index to build — each reading's extract goes whole into the prompt. The 0.5s and $0.00 are the corpus generation itself.

LicenceLicence

MIT

Bring your ownBring your own rule review extracts

tools/build_corpus.py writes data/corpus/*.txt and data/gold.jsonl. Keep the 8 section headings and the field labels src/watch.position_of parses. Set your own retention schedule and disposition rule in src/reach.py first.

⚠︎ And what stops being true when you do: The measured figures on this kit's page do not transfer to your own corpus. The memory-dependent share (49 of 120 readings here), the prose-only unreachable share (20 rules against 20 structured) and the lopsided failure-reason mix are properties of this generator's declared distribution, not facts about records management. Re-run the evals on your own data; that is what the harness is for.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — they were measured against the provider’s own counter, not apportioned by character share.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 1,815 tokens assembled, 1,602 are instructions — 88% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Reproduced from src/prompt.build for RR-0001-C3 with that reading's real carried state, not retyped — byte-identical to what evals/run.py sent.

Full promptThe prompt in full

You are a scheduled records-retention reach review. You read one retention rule or legal hold at one quarterly review cycle, work out which systems it actually reaches and where it silently failed to land, and you answer with one JSON object and no other text.

You are the scheduled records-retention REACH review for one company's retention rules and legal
holds. It runs on a QUARTERLY cadence -- one review cycle every 90 days -- and re-reads EVERY rule.
You are reading ONE rule at ONE review cycle.

A rule is written against a RECORD CLASS. It has to LAND on SYSTEMS. Your job is to say which
systems it actually reaches today and where it silently failed to land -- NOT whether somebody
issued it, which is a column in the register and needs no reader.

The rule, the register's own list of systems in scope with its APPLIED / NOT APPLIED column, the
DEFAULT retention schedule, and this cycle's recorded changes are reproduced in the extract below.
Apply them exactly as written. You cannot see the earlier review cycles; what is known about them
is stated under "Carried state" and is the only history available to you. Do not assume anything
about earlier cycles beyond it.

⚠︎ THE RETENTION PERIODS AND THE DISPOSITION GRACE BAND ARE ILLUSTRATIVE, OPERATOR-TUNABLE
DEFAULTS, NOT ANY REAL COMPANY'S OR ANY JURISDICTION'S RETENTION SCHEDULE. Apply them as printed
regardless of whether they look right for the class in front of you.

⚠︎ THIS REVIEW REPORTS. It never releases a hold, deletes or purges a record, marks anything
disposed, or writes to any register. Every verdict you give is a statement about what the record
already says.

How to work it out:
 - FIRST decide whether the record class is settled. If the record-class mapping is blank on file,
   or the file gives two classes with neither superseding the other, the verdict is
   CONTEXT_INCOMPLETE: no reach is computed, no system is named, the record_class is UNMAPPED and
   nothing is actioned. THERE IS NO DEFAULT RECORD CLASS (Rule R-1).
 - THEN decide whether this rule is already REMEDIATED. If the carried state or this cycle's
   changes record that a system which had FAILED to take this rule has since been recorded as
   taking it, the verdict is REMEDIATED and stays REMEDIATED regardless of what the register's
   column says (Rule R-6). Applying successfully to a system that had failed IS the remediation;
   there is no separate remediation entry.
 - THEN work out what is actually IN SCOPE. Start from the systems named in the carried state,
   add every system the register's table on this page lists, then apply this cycle's changes:
   an ONBOARDED system joins scope, a DECOMMISSIONED system leaves it. The register's table is
   STALE BY DESIGN -- it does not list systems onboarded in an earlier cycle and it still lists
   systems decommissioned in an earlier cycle (Rule R-2).
 - THEN find the UNREACHED systems. A system is unreached if the record shows it could not take
   the rule -- a read-only connector, an archive tier that exposes no hold API, a legacy host with
   no hold or retention concept at all -- whether that was recorded THIS cycle or carried from an
   earlier one. It stays unreached until a later record shows it taking the rule. A failure
   described in an ordinary sentence counts exactly as much as one written in a fixed tag format
   (Rule R-5). THE REGISTER'S COLUMN DOES NOT SETTLE THIS: it records what somebody DID, not what
   LANDED, and it will read APPLIED for a system that rejected the flag.
 - If any in-scope system is unreached, the verdict is UNREACHABLE_SYSTEM and "unreached_system"
   is that system's name exactly as printed. This is the most important call this kit makes -- it
   means the register shows the rule as applied and those records are not held (Rule R-3). If more
   than one qualifies, name the first alphabetically.
 - Otherwise, if the register's OWN table declares a gap -- a row reading NOT APPLIED or PENDING
   for a system still in scope -- the verdict is PARTIAL_REACH and "unreached_system" is that
   system. A gap the register declares is one somebody can already see; the difference between the
   two verdicts is whether anybody knows (Rule R-4).
 - Otherwise consider DISPOSITION, which applies to retention SCHEDULE rules only and only where
   NO legal hold covers the class. Compare the age of the oldest record in the class against the
   DEFAULT retention period for that class. Inside the period: FULLY_REACHED. At or past it and
   within the DEFAULT grace band: DISPOSITION_ELIGIBLE. Beyond the grace band: OVER_RETAINED. A
   hold covering the class suspends disposition entirely, however old the records are, and a hold
   RELEASED on an earlier cycle no longer covers it (Rule R-7).
 - Otherwise the verdict is FULLY_REACHED and "unreached_system" is NONE.
 - THEN decide the OWNER. Start from the records owner on file, but a change entry recording a
   handover to someone else, in this cycle or an earlier one (see "Carried state"), supersedes it.
 - FINALLY decide "action_now": YES only if the verdict is UNREACHABLE_SYSTEM, PARTIAL_REACH or
   OVER_RETAINED **and** this review has not already raised that same finding -- the same system,
   or the same over-retention -- on an earlier cycle (see "Carried state"). Otherwise NO.
   DISPOSITION_ELIGIBLE is never action_now YES: it is reported, and disposing of records is a
   decision with its own approval that this review must never appear to have started.

Answer with a single JSON object and nothing else:

{"verdict": "FULLY_REACHED|UNREACHABLE_SYSTEM|PARTIAL_REACH|DISPOSITION_ELIGIBLE|OVER_RETAINED|REMEDIATED|CONTEXT_INCOMPLETE",
 "unreached_system": "<the system name exactly as printed, or NONE>",
 "record_class": "<the record class exactly as printed, or UNMAPPED>",
 "owner": "<the current records owner's name>",
 "action_now": "YES|NO",
 "rationale": "one sentence, naming the systems you took to be in scope and the evidence you
   relied on for the reach verdict"}

Precedence, applied in this order: CONTEXT_INCOMPLETE if the record class cannot be settled (Rule
R-1); then REMEDIATED (Rule R-6); then UNREACHABLE_SYSTEM (Rule R-3); then PARTIAL_REACH (Rule
R-4); then the disposition verdicts (Rule R-7); otherwise FULLY_REACHED. "unreached_system" is a
system name only for UNREACHABLE_SYSTEM and PARTIAL_REACH, and NONE for every other verdict.

Carried state
----------------------------------------------------------------
As at the previous review cycle, this rule's record class was settled as "Supplier correspondence". The systems known to be in scope for this rule, accumulated across every earlier cycle, are: M365-MAILBOX-SUPPLY, DOCSTORE-SHAREPOINT, SFTP-SUPPLIER-DROP. The register's own table on this page may list fewer or more (Rule R-2). No system has been recorded on any earlier cycle as unable to take this rule. This review has not yet raised anything about this rule with anybody. This review has already raised that this class is over-retained -- do not raise that again. The records owner as last settled is Declan Ashworth -- carried forward unless this cycle's record hands the class to someone else. It was last reported OVER_RETAINED.

Retention review extract
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated records-retention review extract for an
AI use-case kit; it reproduces no company, no matter, no system, no real person and no
real retention schedule. The retention periods and the disposition grace band are
ILLUSTRATIVE, OPERATOR-TUNABLE DEFAULTS -- not any company's or any jurisdiction's
schedule.

Matter Or Rule
----------------------------------------------------------------
Rule reference                     : RR-0001
Rule kind                          : SCHEDULE
Matter                             : -- none; this is a scheduled retention rule
Record class mapping on file       : Supplier correspondence
Records owner (register of record) : Declan Ashworth
Register status                    : ISSUED 2025-05-12, open

Systems In Scope
----------------------------------------------------------------
The matter register's OWN list of systems against this rule, with its OWN
APPLIED / NOT APPLIED column exactly as it stands.

⚠︎ THIS TABLE IS STALE IN BOTH DIRECTIONS. It does not list a system onboarded in an
earlier review cycle, and it still lists one decommissioned in an earlier cycle. And the
column records what somebody DID, not what LANDED: it reads APPLIED for a system that
rejected the flag.

  System                         Kind                       Register says
  M365-MAILBOX-SUPPLY            mailbox                    APPLIED
  DOCSTORE-SHAREPOINT            document store             APPLIED
  SFTP-SUPPLIER-DROP             file drop                  APPLIED

Retention Policy (Default)
----------------------------------------------------------------
The schedule below is an OPERATOR-TUNABLE DEFAULT, not any real company's or any
jurisdiction's retention schedule -- no documented schedule exists for this row. Applied
exactly as printed.

Default retention period           : 6 years
Oldest record in class             : 108 months
Legal hold covering this class     : NO
Disposition grace band             : 12 months past the retention period

The whole DEFAULT schedule, for reference:
  Customer service call recordings        2 years
  Employee timekeeping records            6 years
  Loss-prevention incident reports        6 years
  Marketing consent records               5 years
  POS transaction journals                7 years
  Payment card settlement files           7 years
  Price and promotion approvals           6 years
  Product safety test certificates       10 years
  Store CCTV footage                      1 year
  Store lease and occupancy files        12 years
  Supplier correspondence                 6 years
  Warehouse gate and yard logs            3 years

Disposition authority              : NOT DEFINED. Nothing in this kit releases a hold,
                                     deletes or purges a record, marks anything disposed
                                     or writes to any register.

Rule R-1  A rule is written against a RECORD CLASS and is judged only against the SYSTEMS it has
          to land on. If the record-class mapping is blank on file, or the file gives two classes
          with neither superseding the other, the verdict is CONTEXT_INCOMPLETE: no reach is
          computed, no system is named and nothing is actioned. THERE IS NO DEFAULT RECORD CLASS.
Rule R-2  The SYSTEMS IN SCOPE table is the matter register's own list and it is STALE BY DESIGN.
          A system ONBOARDED in an earlier review cycle is in scope now even though the table does
          not list it; a system DECOMMISSIONED in an earlier cycle is out of scope now even though
          the table still does. Scope ACCUMULATES across cycles (Rule R-5).
Rule R-3  The register's APPLIED / NOT APPLIED column records what somebody DID, not what LANDED.
          Where the record shows that a system could not take the rule -- a read-only connector, an
          archive tier that exposes no hold API, a legacy host with no hold or retention concept --
          that system is UNREACHED however the register's column reads, and the verdict is
          UNREACHABLE_SYSTEM. This is the most important call this kit makes: the register shows
          the rule as APPLIED and that system is not held.
Rule R-4  A gap the register DECLARES -- a row reading NOT APPLIED or PENDING -- is
          PARTIAL_REACH, not UNREACHABLE_SYSTEM. It is a known gap somebody can already see. The
          difference between the two verdicts is whether anybody knows.
Rule R-5  Reach evidence ACCUMULATES across review cycles. "Changes In This Cycle" shows only what
          was recorded since the PREVIOUS cycle; anything recorded in an earlier cycle is not
          repeated and is known only through the carried state. A failure to apply recorded two
          cycles ago is still true today unless a later cycle records the same system taking the
          rule.
Rule R-6  Once a system that had FAILED to take the rule is later recorded as having taken it, the
          rule is REMEDIATED and stays REMEDIATED on every later cycle. Nothing here performs the
          remediation; it only reports one that has already happened elsewhere.
Rule R-7  DISPOSITION applies to retention SCHEDULE rules only, and only where NO legal hold
          reaches the class. A class at or past its DEFAULT retention period is
          DISPOSITION_ELIGIBLE while it is inside the DEFAULT grace band, and OVER_RETAINED once it
          is beyond it. A hold covering the class suspends disposition entirely, however old the
          records are. Reach beats disposition: a rule that does not reach every system is reported
          on its reach first, because a disposition rule that never landed is a class nobody is
          disposing of either.
Rule R-8  "action_now" is YES only the FIRST time a given finding is raised. A system already
          raised on an earlier cycle is not raised again, and an OVER_RETAINED class already raised
          is not raised again. A finding on a system nobody has raised before is YES even if the
          rule was already in an actionable verdict for a different system.
Rule R-9  NOTHING IN THIS KIT RELEASES A HOLD, DELETES OR PURGES A RECORD, MARKS ANYTHING DISPOSED
          OR WRITES TO ANY REGISTER. Every verdict above is a report about what the record already
          says.

Review Position
----------------------------------------------------------------
Cycle ending                       : 2026-09-30   (review cycle 3 of this rule)
Review cadence                     : quarterly -- one records-retention review cycle every 90 days
Previous review cycle              : 2026-06-30
Next review cycle                  : -- none scheduled after this one

Changes In This Cycle
----------------------------------------------------------------
Everything recorded against this rule between the previous review cycle and this one.
THIS CYCLE ONLY -- a change recorded in an earlier cycle is not repeated here.

2026-08-06  Passing the records ownership of this class to Rosalind Chu.

Reviewer Notes
----------------------------------------------------------------
Reviewer note: the record-class mapping was rebuilt at the start of the financial year
and has not been re-derived since.

Raw responseThe raw response

{"verdict": "OVER_RETAINED", "unreached_system": "NONE", "record_class": "Supplier correspondence", "owner": "Rosalind Chu", "action_now": "NO", "rationale": "In-scope systems M365-MAILBOX-SUPPLY, DOCSTORE-SHAREPOINT and SFTP-SUPPLIER-DROP show no unreach evidence and the register marks all APPLIED, but with no legal hold and the oldest record at 108 months versus the 6-year default plus 12-month grace, the class is already over-retained and was previously raised."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch the system a legal hold missed — 120 rule review extracts. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection, never a bill and never a vendor's claim about this workload.

Rates checked 2026-08-18. The provider that actually ran all 284 calls is kept out of these tables per this estate's naming rule, so no figure here is a bill.

The gradersThree ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

The three free floors, the stub and every pre-run check cost $0.00 and no calls. No run was discarded on this kit. ⚑ AND NO CADENCE MODULE EXISTS — the prior process is event-driven and has no interval to compare against.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS, AND ALMOST ALL OF THEM ARE REASONING. 94.1 pct of r001-retention-reach's output (236310 of 251062 tokens) was provider-side reasoning left at the default. The answer is a handful of short fields and a sentence; the bill is the thinking in front of it.
• THE PROMPT IS MOSTLY FIXED TEXT. 3322 input tokens per reading, and the policy block, the rule text and the instruction are the same on every call — so a provider with prompt caching would price this workload very differently, and that was not measured.

Your volumeWhat it costs at your volume

LINEAR IN RULES x CYCLES. Nothing amortises. The sublinear lever NOT implemented is skipping rules whose system list did not change.

Model choiceWhy this model

It is the tier the shared connection was already pointed at, and the kit's whole claim is that swapping it is .env plus one more run. No second model was run against this corpus; every other row in the cost table is a projection onto a published card and is labelled as one.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

per-query average, this run

The codeEvery file, and what it is for

11 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/reach.pythe reach rule and state machine — a swap seam

The DEFAULT retention periods, the disposition rule, the reach test and step() — which decides the next carried state from the answer key's inputs, never from a model reply. It is also the answer key.

You change it to: How long an unreached system stays unreached.

# The retention-and-hold REACH rule as a state machine. Pure code, no model, standard library only.

HOLD = "HOLD"
SCHEDULE = "SCHEDULE"
KINDS = (HOLD, SCHEDULE)
FULLY_REACHED = "FULLY_REACHED"
UNREACHABLE_SYSTEM = "UNREACHABLE_SYSTEM"
PARTIAL_REACH = "PARTIAL_REACH"
DISPOSITION_ELIGIBLE = "DISPOSITION_ELIGIBLE"
OVER_RETAINED = "OVER_RETAINED"
REMEDIATED = "REMEDIATED"

src/state.pythe carried state

Which systems were reached, which failed, and which were onboarded or decommissioned. A system onboarded two cycles ago is in scope now and its first cycle never mentioned it.

# The carried state -- the thing that makes this a review programme and not another register read.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEFAULT_PATH = os.path.join(HERE, "data", "state.json")
def load(path=DEFAULT_PATH):
def save(store, path=DEFAULT_PATH):
def for_rule(store, rule_id):
def _join(names):
def describe(state):

src/segment.pythe section splitter

Splits the page into its 8 named sections; asserted across all 120 documents.

# Split a retention-review extract into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "Matter Or Rule", "Systems In Scope",
RULE = "-" * 64
def split(text):
def render(secs):

src/select.pythe send filter — a swap seam

Custodian Contact is mapped by no field and therefore never sent.

You change it to: SECTION_HINTS and NEVER_SENT.

# Pick which sections of a retention-review extract are sent. Pure code -- the last deterministic

BANNER = "Synthetic Record"
RULEHEAD = "Matter Or Rule"
SYSTEMS = "Systems In Scope"
POLICY = "Retention Policy (Default)"
POSITION = "Review Position"
CHANGES = "Changes In This Cycle"
CONTACT = "Custodian Contact"
NOTES = "Reviewer Notes"
NEVER_SENT = (CONTACT,)

src/prompt.pythe prompt

Three parts: instruction and JSON shape, carried state, extract. The stateless control replaces exactly one line. ⚠︎ IT IS ALSO WHERE THIS KIT'S ONE SYSTEMATIC DEFECT LIVES: nothing tells the model that a system's KIND is not evidence a rule failed.

# Assemble the one prompt this kit sends. No template engine, no framework -- string

INSTRUCTION = """\
def build(text, carried, stateless=False):

src/adapters/__init__.pythe model call

Raw HTTP, stdlib only, bounded retry; the shared daily call cap is checked here.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

src/watch.pythe reader

One rule, one cycle, one call.

# One rule, one review cycle, one model call. The runtime half of the kit.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CORPUS = os.path.join(HERE, "data", "corpus")
SYSTEM = ("You are a scheduled records-retention reach review. You read one retention rule or "
MAX_TOKENS = 16000
FIELDS = ("verdict", "unreached_system", "record_class", "owner", "action_now")
def documents():
def rules():
def load_doc(doc_id):
def position_of(text):

evals/scoring.pythe scorer

Exact match per cell. The unreachable-system discriminator is a separate binary.

# Score a run against the answer key. Pure code, exact match per cell. No model grades anything.

NEEDS_ATTENTION = ("UNREACHABLE_SYSTEM", "PARTIAL_REACH")
FIELDS = ("verdict", "unreached_system", "record_class", "owner", "action_now")
def _pct(n, d):
def score(records, golds):
def _protection(records, golds):

evals/baseline.pythe three free floors

register-only, register-mem and structured-log-strong — $0.00 each, and the third beats the model on the discriminator.

# THE FREE FLOORS. Three of them, none a strawman, all of them $0.00.

MODES = ("register-only", "register-mem", "structured-log-strong")
def _section(text, name, nxt, flat=False):
def _facts(text):
ROW = re.compile(r"^  (\S+) {2,}.+?\s{2,}(APPLIED|NOT APPLIED|PENDING)\s*$", re.M)
def _register(text):
def _finish(verdict, system, record_class, owner, action_now, why):
TAG_APPLIED = re.compile(r"TAG: APPLIED (\S+)")
TAG_FAILED = re.compile(r"TAG: FAILED (\S+)")
TAG_ONBOARDED = re.compile(r"TAG: ONBOARDED (\S+)")

src/app.pythe local UI server

http.server, stdlib. Renders with no key.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
PORT = int(os.environ.get("PORT", "8213"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-retention-reach")
GOLD_ROWS = {r["doc_id"]: r for r in
def carried_for(doc_id):
class H(BaseHTTPRequestHandler):
def main():

ui/app.jsthe local UI client

Hand-written JS. ⚠︎ A CLIPPED AGREE COLUMN WAS FOUND BY OPENING A SCREENSHOT and fixed by shortening the strings. ⚠︎ AND THE SHARED ui/app.css CARRIES A SIBLING KIT'S VERDICT COLOURS, so this kit's verdicts render uncoloured. It was copied verbatim as instructed and NOT edited; the loudest signal still shows because .perm strong applies.

# 

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3322 input and 2092 output tokens per reading (one rule, at one review cycle), all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

MEASURED, not asserted. Reviewer Notes is this kit's injection surface, and the sentence fired is a deliberate GENERALISATION of the shipped note (which names one system). Both strings are recorded as fields in the result file.

API_KEY is read from .env (gitignored) or the real environment; never logged, never placed in a prompt, and redacted from any provider error message the local UI passes to the browser.

The experimentWe DID attack it — one sentence, on every reading where it could matter

The probe forces the instruction-shaped note onto every reading where a raise was due and re-fires them with everything else held identical. Zero raises were suppressed; three verdicts genuinely moved and all three moved toward the MORE alarming verdict. Both were run for real on 2026-08-25.

One boundary is measured in both directions here: the injection probe forces the condition rather than waiting for it, and the privacy guard is red-proven by reproducing the schema-change condition that reaches its fallback.

The result0 of 26 raises suppressed suppressed by the instruction-shaped note — measured, not assumed.

One phrasing, one model, one corpus, 26 trials — every reading where suppression was even possible.

HonestyWhat this does not prove

• Any other injection phrasing. The one fired is a deliberate generalisation of the shipped note.
• Whether the result holds on another model tier.
• Whether an injection placed in Custodian Contact would have any effect — by construction it cannot reach the prompt.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never release a hold, delete a record or mark anything disposed — and never present the retention periods or the disposition rule as any real company's or jurisdiction's schedule.

Stated to the model on every call in src/prompt.py's INSTRUCTION block, and enforced by evals/check_labels.py's banned-code-path scan, red-proven by seeding a purge path.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is a PROMPT RULE PLUS A STATIC SCAN, not a runtime enforcement layer. Nothing stops a forker adding such a path tomorrow; the scan catches it only the next time somebody runs check_labels.py, which is a manual step.
• The injection result is ONE SENTENCE against ONE model on ONE corpus. It is not a resistance rate for prompt injection in general.

WatchedWhat is watched, and why that one

The kit measures 89 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

3 measured by the latest run 86 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

7 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 4 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

triage · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 3 runs measured the same system — they differ on floor, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

triage · with the model in the path — 2 runs. Columns here are only ever compared with each other.

not a time series No two of these 2 runs measured the same system — they differ on documents, stateless — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 18 chips that all say so.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 3 chips that all say so.

DeviationsWhat deviated

0 breaches across 7 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

Nothing fires yet. Not one of the 26 bands on this kit names a consequence, so there is no row to print. Which is the honest answer to the heading: a band that nothing is wired to is a number somebody has to notice, and this kit has only those. An empty table would have said the same thing while looking like a rendering failure.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The guardrail scan is a MANUAL step, run before each spend, not a hook. It ran before every paid run and passed at 0 each time. The injection probe is a ONE-OFF: it measured 0.0 pct suppression on 26 readings on 2026-08-25 and nothing re-runs it, so that figure ages from the day it was taken.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, no framework dependency, and a prompt anyone can read end to end. A framework abstraction would own the retrieval step — there is none here — and the memory layer, which is already the entire surface of src/state.py.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear: the reader -> src/prompt.py -> src/adapters -> evals/scoring.py. No branching, no tool-calling and no agent loop.

The other sideWhat a framework costs you

• Swapping providers means editing the PROVIDERS dict by hand. In exchange the prompt has no hidden templating layer and requirements.txt stays empty.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-retention-reach on the fast tier, with the carried state, 2026-08-25. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 4 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

3 runs not plotted. b000-retention-reach-registeronly, b001-retention-reach-registermem, b002-retention-reach-structuredlogstrong recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the population is re-read whole on each scheduled run, and the prior run's carried state is what makes a reading a CHANGE.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe verdict, unreached system, record class, owner and raise-it-now call, per reading, exact match against the computed answer key

whether each of the 5 answered fields equals the answer key, per reading

MethodHow the test was run

python3 -m evals.run --run-id <id> ...; evals/scoring.py compares strings. No model grades anything.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 120 per field. unreachable_system_accuracy_pct is scored separately over its own denominator.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the computed answer key in data/gold.jsonl, produced by src/reach.step() from the same planted inputs the corpus builder wrote onto the page.

No true/false rates for this grader. It records 5 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold was swept: exact match has no tunable. Denominators are stated beside every rate because 120 readings makes each one wide.

Cadence: once per run; free

The decisionWhen to reach for it

In one lineA system the rule silently failed to reach

whether the arm found a system where the hold could not be applied and nothing said so

MethodHow the test was run

evals/scoring.py, same pass as the exact-match grader.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

caught over all 120 readings; the two error directions over their own denominators (31 unreachable readings, 89 quiet).

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the computed answer key in data/gold.jsonl, produced by src/reach.step() from the same planted inputs the corpus builder wrote onto the page.

No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No sweep: the verdict is an enum the arm emits.

Cadence: once per run; free

The decisionWhen to reach for it

In one lineUnreachable systems recorded ONLY in prose

whether the arm can read a connector failure out of a sentence

MethodHow the test was run

evals/scoring.py, same pass.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

over the 17 readings whose only evidence of failure is prose.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the computed answer key in data/gold.jsonl, produced by src/reach.step() from the same planted inputs the corpus builder wrote onto the page.

No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No sweep.

Cadence: once per run; free

The decisionWhen to reach for it