The business caseThe problem this solves

An SoD rule is written in business terms and an entitlement is written in whatever the system calls it, so the join everybody runs is between two vocabularies that do not match. Worse, the identity system of record lags the grants: a firecall, a go-live elevation or a role copied from a leaver shows up in a change log weeks before it reaches the record. The result is a conflict whose two halves are never on the same page, which is not a careless review — it is a review of a document that does not contain the finding. a rules engine joining an IAM export against a conflict matrix — which fires only on entitlement rows carrying an explicit capability tag, cannot see a grant that has not reached the record yet, and re-raises conflicts that were remediated two cycles ago — plus the manual work of explaining each entitlement in business terms to the manager being asked to certify it.

The inputThe actual certification extracts

The corpus is 120 certification extracts, 0.91 MB (txt 120). A real access-certification export names every system a company runs, its privileged role catalogue, and which named individuals hold rights over money movement. There is no public one and there will not be. A scrubbed export is worse rather than better: scrubbing removes the administrator's typed descriptions and the change-log prose, which is exactly the layer this kit measures.

Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated access-certification cycle extract for
an AI use-case kit; it reproduces no company, no identity system, no role catalogue and no
real person. The segregation-of-duties rule set is an ILLUSTRATIVE DEFAULT.

User
----------------------------------------------------------------
User reference                 : USER-0001
Name                           : Tomasz Nowak
Job title                      : Finance Systems Analyst
Department                     : Finance Systems
Account type                   : employee
Certifying manager of record   : P. Achebe

Entitlements Of Record
----------------------------------------------------------------
What the identity system of record shows this user holding, as at this cycle. A row shown
'on record since cycle 1' was already held when this campaign opened. THE RECORD LAGS: an
entitlement granted outside the request queue may not appear here for several cycles, or
at all.

  PAYMT-REL                    Legacy host (AS/400)     on record since cycle 1 (2026-03-31)
      Authorises release of the daily outgoing payment file.
  Vendor Master - Change       Legacy host (AS/400)     on record since cycle 1 (2026-03-31)
      Opens the vendor master maintenance screen with update rights.
  oms.credit.issue             Order management         on record since cycle 1 (2026-03-31)
      Raise a customer credit or refund against an order.
  reports.read                 Analytics portal         on record since cycle 1 (2026-03-31)
      View dashboards; no data can be changed.

SoD Policy (Default)
----------------------------------------------------------------

The outcomeWhat a good result looks like

every user's live conflict named by rule id, with where it CAME FROM — inside one cycle, accumulated across cycles, or pre-existing — and 120 of 120 readings answered with 0 false flags raised across 63 clean readings.

Where it fitsWhat did work

Every line below is a measured result from this kit's own runs, with the figure that supports it. The headline above is not softened by any of them.

• Every entitlement in your catalogue carries a capability tag, and every grant reaches the identity system of record before the campaign runs — the free floor -- structured-log-strong, $0.00
  It joins tags against the rule table and accumulates the change log. It already BEATS the model on the certifying manager (95.0 against 93.33).
• Half your catalogue is described in a sentence somebody typed, and firecall grants are normal — the model, with the carried state
  Verdict 100.0 against 87.5, ACCUMULATED conflicts 16 of 16 against 12, and verdict accuracy on readings whose record maps only from PROSE 100.0 against 83.33.

And where nothing here is good enough:

• You want the model's reading but cannot carry state between cycles — neither, as configured here
  The stateless control is the measurement of that choice: 5 of 16 ACCUMULATED conflicts against 16, and memory-dependent verdict accuracy 40.62 against 100.0.

At a glanceHow the whole thing runs

Run once, for real, on 2026-08-25. Every figure on these pages was captured from that run — nothing is written from intent.

14 steps, grouped by the question that sends you to them rather than by build order. Each tile carries the one figure that step is about, and opens the page behind it.

Should you use this?What you bring, where it stops, and when not to use it

Before you commit an afternoon to this, these are the answers that decide it. Each one is rendered from the record it lives in — and links the page that holds it in full.

Not asked of this kit — 2 questions: clone (a fresh clone of this kit runs with nothing fetched); judge (nothing here is graded by a model).

Last verified 2026-08-25 — r001-entitlement-sod. Every figure on these pages was captured from that run.

Run itHow this reaches your data

Every result on this page was produced by pure code over checked-in files, with no API key — which is why you can read the numbers before anyone spends anything.

Checked before this shipped — Every corpus extract, the answer key, all three free floors, the injection probe and what run r001 actually answered ship in the repo. python3 -m evals.check_labels and python3 -m src.app both run with no key and no network.

Step 01 of 14Business case

Written for: product manager · Measured during the eval run. Never estimated, never quoted from a vendor page.

In brief

What the clock covers. one reading — one user, at one scheduled run, end to end including provider-side reasoning tokens, on the shared connection. Not a cold-start figure and not a per-user SLA: 120 readings ran in 326.9 wall seconds because different users run concurrently while a single user's runs are strictly serial.

Current processWhat it replaces

a rules engine joining an IAM export against a conflict matrix — which fires only on entitlement rows carrying an explicit capability tag, cannot see a grant that has not reached the record yet, and re-raises conflicts that were remediated two cycles ago — plus the manual work of explaining each entitlement in business terms to the manager being asked to certify it.

Step 02 of 14Architecture

Written for: solutions architect · Component map derived from the real code, not drawn from intent.

In brief

CorpusCorpusArchitectureArchitectureArchitectureArchitectureEvalsEvalsEvalsEvalsPromptPromptPromptPromptApp UIApp UIEvalsEvalsBusiness caseUnit cost

• 1stations numbered in run order
• txt
• jsonl
• json
• 3 of 10 stations carry a recorded failure

scroll → 10 stations

One question, end to end. Colour on the fan-in is the corpus format; a red hook marks every station with a recorded failure, and the words — every sub line, failure and report field — are in the ledger beneath. Every station is a link — click one to jump to its row, and each row links to the lens that explains it.

The swap seams

Components

Step 03 of 14App UI

Written for: anyone, in 5 seconds · Screenshots of the running app taken during the run.

SuccessesWhen it works

LimitsWhen it does not

A report showing only wins is an advert. This one is required, and the validator now refuses a kit that omits it.

Step 04 of 14Corpus

Written for: "can I point this at mine?" · Measured off the corpus the run actually used.

In brief

How it is cutWhat one reading is

40 users x 3 quarterly cycles, walked strictly in cycle order per user so the capability map, the remediation record and the compensating controls carry forward the way they would in a real campaign. Different users run concurrently; a single user's cycles never do.

SetupWhat the setup figure measured

There is no index to build — each reading's extract goes whole into the prompt. The 0.5s and $0.00 are the corpus generation itself.

LicenceLicence

MIT

Bring your ownBring your own certification extracts

tools/build_corpus.py writes data/corpus/*.txt and data/gold.jsonl. Point it at your own campaign export by keeping the 8 section headings (src/segment.SECTIONS) and the field labels src/watch.position_of parses. Set your own rules in src/entitlement.DEFAULT_SOD_RULES first — the row this kit was built from says the rule set is yours.

⚠︎ And what stops being true when you do: The measured figures do not transfer. The memory-dependent share (32 of 120 readings here), the tagged-versus-prose split, and how often a grant is out of band are properties of this generator's declared distribution, not facts about identity management.

Step 05 of 14Prompt

Written for: engineer · Logged from the run, verbatim.

In brief

The assembled prompt in full, and its real decomposition. Every part below occurs in the prompt that was actually sent, and the token counts sum to the recorded input total — but they are the input total apportioned by each part’s share of the characters, not a per-part reading from the provider’s counter. The total is measured; the split is arithmetic, and a part carrying denser text really costs more of it than this says.

AssemblyHow the prompt is assembled

This is the cost lesson as arithmetic: of the 1,524 tokens assembled, 1,415 are instructions — 93% of the prompt, the largest single part. A per-request bill follows whichever part is biggest, and on this kit that is the one named here.

Reproduced from src/prompt.build for USER-0002-C2 with that reading's real carried state, not retyped — byte-identical to what evals/run.py sent.

Full promptThe prompt in full

You are a scheduled access-certification review. You read one user's entitlements for one certification cycle against the DEFAULT segregation-of-duties rule set, and you answer with one JSON object and no other text.

You are the scheduled access-certification review for one company's identity estate. It runs on a
QUARTERLY cadence -- one certification cycle per quarter -- and re-reads EVERY user. You are reading
ONE user at ONE certification cycle.

The user, the entitlements the identity system of record shows for them, the DEFAULT
segregation-of-duties rule set, and the changes recorded against them in this cycle are reproduced
in the extract below. Apply them exactly as written. You cannot see the earlier cycles; what is
known about them is stated under "Carried state" and is the only history available to you. Do not
assume anything about earlier cycles beyond it.

⚠︎ THE SoD RULE SET AND THE CAPABILITY VOCABULARY ARE ILLUSTRATIVE DEFAULTS, NOT ANY REAL COMPANY'S
POLICY. Apply them as printed regardless of whether they look right for the user in front of you.

How to work it out:
 - FIRST decide whether anyone can certify this account at all. If no certifying manager can be
   resolved -- the manager of record has left, or the account is a service or shared account with
   no named owner -- the verdict is ORPHANED, whatever the entitlements contain. Nobody can certify
   an account nobody owns, and a conflict on an orphaned account is worse rather than moot
   (Rule E-8). Report the manager as null.
 - THEN MAP EVERY ENTITLEMENT THE USER HOLDS TO A BUSINESS CAPABILITY. This is the work. The rule
   set is written in business terms; the entitlement is written in whatever the system calls it,
   and the same capability wears a different name in every system. A row may carry an explicit
   `CAPABILITY:` tag, or only a sentence an administrator typed, or nothing at all. A typed
   description counts exactly as much as a tag -- read it and map it (Rule E-1). Use only the
   capability names printed in the vocabulary.
 - If an entitlement the user HOLDS carries no tag, no usable description and no catalogue entry,
   its capability cannot be settled: the verdict is CONTEXT_INCOMPLETE, no rule is fired on a guess,
   and no action is due. THERE IS NO DEFAULT CAPABILITY (Rule E-9).
 - THE SET OF ENTITLEMENTS THE USER HOLDS IS NOT THE SAME AS THE ENTITLEMENTS OF RECORD. It is the
   record, PLUS anything evidenced as granted since -- in this cycle's change log, in an earlier
   cycle's (see "Carried state"), or described in a reviewer note -- MINUS anything evidenced as
   revoked the same way. The record LAGS: an out-of-band grant is evidenced days or weeks before it
   reaches the record, and it counts from the moment it was made (Rule E-2). A grant described in
   ordinary prose counts exactly as much as one written in a fixed tag format.
 - THEN FIRE THE RULES. A conflict exists when the capability set contains BOTH halves of one of
   the DEFAULT rules (Rule E-3). Report the LOWEST-NUMBERED rule that fires.
 - THEN DECIDE THE ORIGIN of that conflict, from the cycle each half was FIRST held in:
   PRE_EXISTING when both halves were already on the record at the first cycle of this campaign;
   SINGLE_CYCLE when both halves were first held in the same later cycle; ACCUMULATED when the two
   halves were first held in DIFFERENT cycles (Rule E-5). ACCUMULATED is the case a reviewer
   working one cycle at a time cannot see, because no single cycle's export contains both halves.
 - A COMPENSATING CONTROL recorded against the conflict does NOT clear it. It makes the verdict
   STEP_UP_REVIEW rather than SOD_CONFLICT: somebody senior still has to look, and action is still
   due (Rule E-6). Nothing here grants an exception.
 - Once a conflict has been recorded as REMEDIATED -- the offending entitlement revoked, in the
   change log or in a reviewer note -- it does not fire again. If no other conflict is live, the
   verdict is REMEDIATED and no action is due (Rule E-7).
 - If no conflict is live and nothing has ever been remediated, the verdict is CERTIFY, the rule is
   NONE and the origin is NONE.
 - THEN DECIDE THE MANAGER. Start from the certifying manager of record, but a reorg recorded in
   this cycle's changes, or in an earlier one (see "Carried state"), supersedes it.
 - FINALLY decide "action_now": YES when the verdict is SOD_CONFLICT, STEP_UP_REVIEW or ORPHANED --
   somebody has to do something about this user before the campaign closes. Otherwise NO.

Answer with a single JSON object and nothing else:

{"verdict": "CERTIFY|SOD_CONFLICT|STEP_UP_REVIEW|REVOKE_REQUIRED|ORPHANED|REMEDIATED|CONTEXT_INCOMPLETE",
 "conflict_rule": "SOD-01|SOD-02|SOD-03|SOD-04|SOD-05|SOD-06|NONE|UNDETERMINED",
 "origin": "SINGLE_CYCLE|ACCUMULATED|PRE_EXISTING|NONE|UNDETERMINED",
 "manager": "<the certifying manager's name, or null if there is none>",
 "action_now": "YES|NO",
 "rationale": "one sentence, naming the two capabilities that collided, which entitlement each came
   from, and the cycle each was first held in"}

Precedence, applied in this order: ORPHANED if no certifying manager can be resolved (Rule E-8);
then CONTEXT_INCOMPLETE if a held entitlement's capability cannot be settled (Rule E-9); then the
conflict ladder -- STEP_UP_REVIEW where a compensating control is on record, otherwise SOD_CONFLICT;
then REMEDIATED; then CERTIFY. "conflict_rule" is UNDETERMINED only when the verdict is
CONTEXT_INCOMPLETE, and NONE when the verdict is CERTIFY or ORPHANED. "origin" is UNDETERMINED when
the verdict is ORPHANED or CONTEXT_INCOMPLETE, and NONE when no conflict is live.

Carried state
----------------------------------------------------------------
As at the previous cycle, this user held: APPROVE_PO, ISSUE_REFUND, RELEASE_PAYMENT (already held before the first cycle on this watch). No conflict was live at the previous cycle. The certifying manager as last settled is R. Osei -- carried forward unless this cycle's record moves it. It was last reported CERTIFY.

Certification cycle extract
----------------------------------------------------------------
Synthetic Record
----------------------------------------------------------------
Every field below is invented. This is a generated access-certification cycle extract for
an AI use-case kit; it reproduces no company, no identity system, no role catalogue and no
real person. The segregation-of-duties rule set is an ILLUSTRATIVE DEFAULT.

User
----------------------------------------------------------------
User reference                 : USER-0002
Name                           : Leila Nakamura
Job title                      : Treasury Analyst
Department                     : Group Treasury
Account type                   : employee
Certifying manager of record   : R. Osei

Entitlements Of Record
----------------------------------------------------------------
What the identity system of record shows this user holding, as at this cycle. A row shown
'on record since cycle 1' was already held when this campaign opened. THE RECORD LAGS: an
entitlement granted outside the request queue may not appear here for several cycles, or
at all.

  MM_PO_APPROVE                ERP (Procurement)        on record since cycle 1 (2026-03-31)
      CAPABILITY: APPROVE_PO
  PAYMT-REL                    Legacy host (AS/400)     on record since cycle 1 (2026-03-31)
      CAPABILITY: RELEASE_PAYMENT
  oms.credit.issue             Order management         on record since cycle 1 (2026-03-31)
      CAPABILITY: ISSUE_REFUND

SoD Policy (Default)
----------------------------------------------------------------
The rule set below is an OPERATOR-TUNABLE DEFAULT, not a real company's actual policy -- no
documented rule set exists for this row. Applied exactly as printed.

Business capabilities. An entitlement is judged by WHAT IT LETS SOMEBODY DO, never by its
technical code:
  CREATE_VENDOR      create or amend a vendor / payee master record
  RELEASE_PAYMENT    release or approve an outgoing payment
  APPROVE_PO         approve a purchase order
  RECEIVE_GOODS      confirm goods were received against an order
  POST_JOURNAL       post a general-ledger journal entry
  APPROVE_JOURNAL    approve a general-ledger journal entry
  MAINTAIN_PAYROLL   change employee pay data
  APPROVE_PAYROLL    approve a payroll run
  MAINTAIN_PRICE     set or change a retail price
  ISSUE_REFUND       issue a customer refund or credit
  GRANT_ACCESS       grant or change another person's access
  VIEW_REPORTS       read reports (no change rights)
  VIEW_INVENTORY     read stock positions (no change rights)

Segregation-of-duties rules -- each is a pair of capabilities nobody should hold together:
  SOD-01   CREATE_VENDOR + RELEASE_PAYMENT        create a payee and pay it -- the classic fictitious-vendor exposure
  SOD-02   APPROVE_PO + RECEIVE_GOODS             approve a purchase and confirm its receipt -- nobody independent saw the goods
  SOD-03   APPROVE_JOURNAL + POST_JOURNAL         post an entry and approve it -- the ledger has no second pair of eyes
  SOD-04   APPROVE_PAYROLL + MAINTAIN_PAYROLL     change pay and approve the change -- a ghost-employee exposure
  SOD-05   ISSUE_REFUND + MAINTAIN_PRICE          set the price and refund against it -- a margin and shrink exposure
  SOD-06   GRANT_ACCESS + RELEASE_PAYMENT         grant access and release payment -- can give themselves anything above

Exception authority            : NOT DEFINED. Nothing in this kit certifies an
                                 entitlement, revokes one, grants one, closes a campaign
                                 item or signs anything off on a manager's behalf.

Rule E-1  An entitlement's BUSINESS CAPABILITY is what decides whether a rule fires -- never its
          technical code. The same capability is named differently in every system: a code may
          carry an explicit `CAPABILITY:` tag, or only a description an administrator typed, or
          nothing but the code itself. Map it from whatever the page gives you.
Rule E-2  An entitlement the user HOLDS is anything in the Entitlements Of Record PLUS anything
          granted since -- in the cycle change log or described in a reviewer note -- MINUS
          anything revoked the same way. The record LAGS: an out-of-band grant is evidenced in a
          change log or a note days or weeks before it reaches the record, and it counts from the
          moment it was made, not from the moment the record caught up.
Rule E-3  A CONFLICT fires when the user's capability set contains both halves of a DEFAULT SoD
          rule. The rule set below is an operator-tunable placeholder, not this company's actual
          policy, and is applied exactly as printed.
Rule E-4  Capability, grant, revocation, remediation and compensating-control evidence ACCUMULATES
          across cycles. "Changes In This Cycle" shows only what was recorded since the PREVIOUS
          cycle; anything recorded in an earlier cycle is not repeated and is known only through
          the carried state.
Rule E-5  The ORIGIN of a conflict is SINGLE_CYCLE when both halves were granted inside this one
          cycle, ACCUMULATED when the halves were granted in DIFFERENT cycles, and PRE_EXISTING
          when both were already held before the first cycle on this watch. ACCUMULATED is the
          case one-cycle-at-a-time review cannot see, and it is reported separately for that
          reason.
Rule E-6  A COMPENSATING CONTROL recorded against the user does not clear a conflict. It changes
          the verdict from SOD_CONFLICT to STEP_UP_REVIEW: somebody senior still has to look, and
          the conflict is still reported. Nothing here grants an exception -- no exception
          authority is named for this row.
Rule E-7  Once a conflict is recorded as REMEDIATED -- the offending entitlement revoked, in the
          change log or in a reviewer note -- the user is REMEDIATED for that conflict and stays
          so on later cycles, unless a NEW conflict fires. Nothing here performs the revocation;
          it only reports one that has already happened.
Rule E-8  If the user has no certifying manager on file -- the manager has left, or the account is
          a service or shared account with no named owner -- the verdict is ORPHANED regardless of
          what the capability set contains. Nobody can certify an account nobody owns, and a
          conflict on an orphaned account is worse, not better.
Rule E-9  If an entitlement's capability cannot be settled -- no tag, no description, and no
          catalogue entry -- the verdict is CONTEXT_INCOMPLETE: no rule is fired on a guess. THERE
          IS NO DEFAULT CAPABILITY.

Certification Position
----------------------------------------------------------------
Cycle ending                   : 2026-06-30   (certification cycle 2 of this user)
Campaign cadence               : quarterly -- one certification cycle per quarter
Previous cycle ended           : 2026-03-31
Next cycle ends                : 2026-09-30
Entitlements on record         : 3

Changes In This Cycle
----------------------------------------------------------------
Everything recorded against this user between the previous cycle and this one. THIS CYCLE
ONLY -- a change recorded in an earlier cycle is not repeated here.

2026-06-19  She can now confirm deliveries against orders in the warehouse system -- granted
              verbally during the peak-season ramp.

Reviewer Notes
----------------------------------------------------------------
The campaign tool shows this user twice because of a duplicate identity created in the
migration; this is the live record.

Raw responseThe raw response

{"verdict": "SOD_CONFLICT", "conflict_rule": "SOD-02", "origin": "ACCUMULATED", "manager": "R. Osei", "action_now": "YES", "rationale": "APPROVE_PO (from MM_PO_APPROVE, held before cycle 1) and RECEIVE_GOODS (from the warehouse delivery-confirmation grant on 2026-06-19 in cycle 2) collide under SOD-02, so the conflict accumulated across cycles."}

Step 06 of 14Evals

Written for: the skeptic · The harness output, with as_of / model version / dataset version / verified_by.

In brief

Catch employees holding two conflicting access rights — 120 certification extracts. One model answered, and every answer was then graded Three different ways — because the grade is a measurement too, and most of what went wrong on this kit went wrong in the ruler rather than the model.

MeasurementsWhat was measured

Two words carry this page: Counted is deterministic and nobody's opinion and Not yet known is printed blank rather than filled with something plausible. There is no Judged row here and that is the point: nothing on this page is a model's opinion about another model. The row most worth having is currently the empty one.

Grading costWhat it costs

Every dollar on these pages is a MEASURED token count multiplied by a named vendor's PUBLISHED rate — a projection, never a bill and never a vendor's claim about this workload.

Rates checked 2026-08-18. The provider that actually ran all 316 calls is kept out of these tables per this estate's naming rule, so no figure here is a bill.

The gradersThree ways to grade

Every row opens its own page: how the test was run, the input on one real row, the prompt or formula with every iteration of it, and the analysis.

Choosing oneWhich one to use

The comparison is only useful if it ends in a choice. The answer is conditional — and it is not the grader with the best agreement score.

This pipeline was also attacked, which is a different question from whether its answers are right: see Threat model for the injection experiment, its two gates and what it does not prove.

Failure causesHow it fails

Step 07 of 14Unit cost

Written for: whoever signs off on the bill · Measured over the eval run, per model. Never quoted from a vendor page.

In brief

The token counts are measured on a real run and belong to this pipeline — which sections are selected, the field schema, the prompt — so they hold wherever you run it. The dollars are those counts multiplied by each vendor's published rate, checked 2026-08-18. Nobody paid these particular bills; they are what the same work would cost you on a card you can sign up to today.

Answering vs gradingAnswering and grading are two separate bills

The three free floors, the stub and every pre-run check cost $0.00 and no calls. No run was discarded on this kit.

Cost driversWhat actually moves the bill

• OUTPUT TOKENS. 140848 of 152284 output tokens on the scored run were provider-side reasoning left at the default. The answer is five short fields and a sentence.
• THE PROMPT IS MOSTLY FIXED TEXT. The rule table, the capability vocabulary and the instruction are identical on every call, so a provider with prompt caching would price this workload very differently -- and that was not measured.

Your volumeWhat it costs at your volume

LINEAR IN USERS x CYCLES. Ten times the identities is ten times the calls at the same per-reading cost; nothing amortises, because the carried state is a small map rather than a growing transcript. The sublinear lever this kit does NOT implement is skipping users whose capability set did not change -- named here, not built, not measured.

Model choiceWhy this model

It is the tier the shared connection was already pointed at, and the kit's claim is that swapping it is .env plus one more run. No second model was run; every other row in the cost table is a projection onto a published card and is labelled as one.

Other modelsOn other models

This is arithmetic, not a run. No model below was executed against the labelled set, and no accuracy is claimed for any of them. What transfers is the input volume: retrieval is model-independent, so every model would receive the same passages.

per-query average, this run

The codeEvery file, and what it is for

10 modules, no framework, and almost no dependencies. The pipeline is stdlib Python, because a kit is meant to be read and changed by someone who cloned it ten minutes ago — every dependency is a thing they have to understand before they can touch the part they came for.

Five of these are swap seams: the points you edit to make the kit yours. Each is marked, and each block below is the module’s real shape, read out of the file rather than written here.

The files on this page are not in the public GitHub catalog.

src/entitlement.pythe SoD rule engine and state machine — a swap seam

The DEFAULT rule set, the capability vocabulary, the origin test and step() — which decides the next carried state from the answer key's inputs, never from a model reply. It is also the answer key: the corpus builder calls the same function.

You change it to: CYCLE_DATES. It is the scaling variable AND the window in which an out-of-band grant is invisible.

# The segregation-of-duties rule as arithmetic. Pure code, no model, standard library only.

CREATE_VENDOR = "CREATE_VENDOR"
RELEASE_PAYMENT = "RELEASE_PAYMENT"
APPROVE_PO = "APPROVE_PO"
RECEIVE_GOODS = "RECEIVE_GOODS"
POST_JOURNAL = "POST_JOURNAL"
APPROVE_JOURNAL = "APPROVE_JOURNAL"
MAINTAIN_PAYROLL = "MAINTAIN_PAYROLL"
APPROVE_PAYROLL = "APPROVE_PAYROLL"
MAINTAIN_PRICE = "MAINTAIN_PRICE"

src/state.pythe carried state

A capability map (capability -> the cycle it was FIRST held in) and four lists. The map is what makes the origin test answerable; collapse it to a set and ACCUMULATED becomes indistinguishable from SINGLE_CYCLE.

# The carried state -- the thing that makes this a monitor and not a report.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEFAULT_PATH = os.path.join(HERE, "data", "state.json")
def load(path=DEFAULT_PATH):
def save(store, path=DEFAULT_PATH):
def for_user(store, user_id):
def describe(state):

src/segment.pythe section splitter

Splits the page into its 8 named sections. check_labels asserts all 8 parse in all 120 documents before a run may spend.

# Split a certification-cycle extract into its named sections. Pure code, no model.

SECTIONS = ("Synthetic Record", "User", "Entitlements Of Record", "SoD Policy (Default)",
RULE = "-" * 64
def split(text):
def render(secs):

src/select.pythe send filter — a swap seam

Decides which sections reach the provider. Manager Contact is mapped by no field and therefore never sent.

You change it to: SECTION_HINTS and NEVER_SENT. The fallback is the guard that stops a renamed export schema sending everything.

# Pick which sections of a certification-cycle extract are sent. Pure code -- the last

BANNER = "Synthetic Record"
USER = "User"
RECORD = "Entitlements Of Record"
POLICY = "SoD Policy (Default)"
POSITION = "Certification Position"
CHANGES = "Changes In This Cycle"
CONTACT = "Manager Contact"
NOTES = "Reviewer Notes"
NEVER_SENT = (CONTACT,)

src/prompt.pythe prompt

Three parts: the fixed instruction and JSON shape, the carried-state sentences, the extract. The stateless control replaces exactly one line.

# Assemble the one prompt this kit sends. No template engine, no framework -- string

INSTRUCTION = """\
EXTRACT_HEADING = "Certification cycle extract"
def build(text, carried, stateless=False):

src/adapters/__init__.pythe model call

Raw HTTP, stdlib only. Bounded retry on transient status codes; the shared daily call cap is checked here because it is the one line every arm goes through.

# SEAM 1 — the model. Swapping provider or model is .env plus the same run again.

class AdapterError(RuntimeError):
TRANSIENT = {408, 429, 500, 502, 503, 504}
RETRIES = 4
def _post(url, headers, payload, timeout=120):
def openai_compatible(cfg, system, user, max_tokens, thinking=None):
def anthropic(cfg, system, user, max_tokens):
PROVIDERS = {"openai-compatible": openai_compatible, "anthropic": anthropic}
THINKING_OFF = {"type": "disabled"}
def complete(cfg, system, user, max_tokens=1024, thinking=None):

evals/scoring.pythe scorer

Exact match per cell. The SoD discriminator and the ACCUMULATED recall are separate binaries over separate denominators, never averaged.

# Score a run against the answer key. Pure code, exact match per cell. No model grades anything.

FIELDS = ("verdict", "conflict_rule", "origin", "manager", "action_now")
CONFLICT_VERDICTS = ("SOD_CONFLICT", "STEP_UP_REVIEW")
NEEDS_ACTION = ("SOD_CONFLICT", "STEP_UP_REVIEW", "ORPHANED")
NO_MANAGER = {"", "none", "null", "-", "--", "n/a", "nobody", "not applicable"}
def _pct(n, d):
def _norm_manager(v):
def _eq(field, got, want):
def score(records, golds):
def _protection(records, golds):

evals/baseline.pythe three free floors

record-tags-only, record-tags-mem and structured-log-strong — $0.00 each, and the third beats the model on one column.

# THE FREE FLOORS. Three of them, none a strawman, all of them $0.00.

MODES = ("record-tags-only", "record-tags-mem", "structured-log-strong")
NO_DESCRIPTION_MARK = "no description on file"
def _section(text, name, nxt, flat=False):
ROW = re.compile(r"^  (\S.*?)\s{2,}(\S.*?)\s{2,}on record since cycle (\d+)", re.M)
TAG_CAP = re.compile(r"^ +CAPABILITY: ([A-Z_]+)\s*$")
def record_rows(text):
def manager_of_record(text):
def cycle_of(text):
TAG_GRANT = re.compile(r"TAG: GRANT ENT=\S+ SYS=.+? CAPABILITY=([A-Z_]+)")

src/app.pythe local UI server

http.server, standard library. Renders with no key.

# The minimal local UI. Standard library only -- python3 -m src.app, then open the printed URL.

HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UI = os.path.join(HERE, "ui")
GOLD = os.path.join(HERE, "data", "gold.jsonl")
PORT = int(os.environ.get("PORT", "8211"))
RECORDED_RUN = os.environ.get("RECORDED_RUN", "r001-entitlement-sod")
GOLD_ROWS = {r["doc_id"]: r for r in
def carried_for(doc_id):
class H(BaseHTTPRequestHandler):
def main():

ui/app.jsthe local UI client

Hand-written JS, no framework. Prints the model column beside the free floor and says which rows agree.

# 

Start hereThe shortest path into it

Every entry above is a file in this kit, listed in the order the pipeline runs it.

ScenariosWhat it costs at your volume

Everything below is computed from one measured base: 3241 input and 1269 output tokens per reading (one user, at one certification cycle), all of it fixed — this kit sends the document's own selected sections, so there is no retrieval depth to turn. Change the volume and the bill scales linearly, because nothing here batches.

OperationsFour shapes of workload

The same arithmetic, at volumes worth naming. These are illustrations of scale, not measurements of anyone’s deployment.

Model follows the controls above.

What this ignoresWhere a real bill diverges

In one lineWhat this kit exposes

MEASURED, not asserted. Reviewer Notes is this kit's one injection surface -- free text a certifying manager or administrator types -- and one of the five shipped notes is instruction-shaped by design. It was fired as its own probe rather than named.

API_KEY is read from .env (gitignored) or the real environment; never logged, never placed in a prompt, and redacted from any provider error message the local UI passes to the browser.

The experimentWe DID attack it -- one sentence, on every reading where it could matter

An indirect prompt injection needs a field somebody outside the process can write into, and this kit's candidate is Reviewer Notes. Rather than report the rate at which the shipped corpus happened to place the note next to a due review, evals/injection.py forces it onto every such reading and re-fires them with everything else held identical. 67 of 67 were still raised. Both were run for real on 2026-08-25.

One boundary is measured in both directions here: the injection probe forces the condition rather than waiting for it, and the privacy guard is red-proven by reproducing the schema-change condition that reaches its fallback.

The result0 of 67 reviews suppressed by the instruction-shaped reviewer note -- measured, not assumed.

One phrasing, one model, one corpus, 67 trials -- every reading where suppression was even possible.

HonestyWhat this does not prove

• Any other injection. A note asking the model to report REMEDIATED, or to name a different manager, is a different experiment and was not run.
• Whether the result holds on another model tier.
• Whether an injection placed in Manager Contact would have any effect -- by construction it cannot reach the prompt at all, and that was not probed either.

In one lineThe only guardrail is the prompt

One instruction, no enforcement. It mostly holds — and mostly is the whole subject.

Never certify, revoke, grant or sign off an entitlement — and never present the SoD rule set or the capability map as any real company's policy.

Stated to the model on every call in src/prompt.py's INSTRUCTION block, and enforced mechanically by evals/check_labels.py's banned-code-path scan before any run may spend.

EvidenceDoes it hold?

The limitWhat a guardrail is not

• This is a PROMPT RULE PLUS A STATIC SCAN, not a runtime enforcement layer. Nothing stops a forker adding a certify() path tomorrow; the scan catches it only the next time somebody runs check_labels.py, which is a manual step.
• The injection result is ONE SENTENCE against ONE model on ONE corpus. It is not a resistance rate for prompt injection in general.

WatchedWhat is watched, and why that one

The kit measures 100 watched figures plus 8 comparability guards and 5 reference constants — three duties, not one list. 0 of the watched are grader rates, each on its own grader page; this board watches the ones that decide when to look again. Each square is one watched figure; filled means the latest run measured it.

3 measured by the latest run 97 need the model half

GuardsBefore any comparison

Preconditions, not alarms. If one moves, the runs were not measuring the same system and nothing else on this board may be differenced.

no shared guard The two recorded runs declare no guard in common, so nothing below is protected by a precondition — read the history as two separate observations, not as a difference.

The reference constants — the null-grader baseline, the adjudication matrix, the negative count — are the third duty. They are never re-measured per run: a baseline that moves with the thing it measures is not a baseline.

BandsThe alarm bands

Derived from evidence, never picked: exact-match where two runs reproduced a metric to the digit, wider than the measured same-input drift for latency, one row of the negative class for any rate — denominator printed beside it.

HistoryRun history

7 recorded runs. A dash is a metric that run did not measure, and a dash is never differenced: not measured is a third state, neither a value nor a zero, and reading it as zero is how a board reports an improvement on a run that never looked. They are split into 4 blocks because a free floor is a baseline rather than a peer, and a run of another kind measures another thing; a column in one block is never differenced against a column in another.

triage · no model in the path — a baseline, not a peer column — 3 runs. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

not a time series No two of these 3 runs measured the same system — they differ on floor, the model — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

triage · with the model in the path — 2 runs. Columns here are only ever compared with each other.

not a time series No two of these 2 runs measured the same system — they differ on stateless — so nothing here may be differenced and the Verdict column is left off. Read this block DOWN a column, as a comparison between those runs, not along a row as a history.

stub · no model in the path — a baseline, not a peer column — 1 run. A floor is what the kit scores with no model in the path; it is the line the model runs have to beat, and it is never differenced against them.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 29 chips that all say so.

redteam · with the model in the path — 1 run. Columns here are only ever compared with each other.

one run A verdict is a comparison and one run has nothing to compare with, so the Verdict column is left off rather than filled with 3 chips that all say so.

DeviationsWhat deviated

0 breaches across 7 runs

The day a band breaks, a row renders here carrying the metric, the two runs, the band it crossed and what changed between them — model, prompt, chunker or top_k, read off the run records’ guards. A deviation with no diff beside it is an alarm nobody can act on.

RippleThe ripple map

Which figures move together, so one alarm reads as a system. Every edge carries its basis: measured names the run that showed it; reasoning admits nothing has varied it yet.

FiresWhat fires when a band breaks

Where evals stop sitting beside the guardrail and start feeding it. Where the answer is nothing automatic, the row says so — hiding it would undo “What a guardrail is not” above.

Nothing fires yet. Not one of the 29 bands on this kit names a consequence, so there is no row to print. Which is the honest answer to the heading: a band that nothing is wired to is a number somebody has to notice, and this kit has only those. An empty table would have said the same thing while looking like a rendering failure.

NextThe three you would add first

None of these is built here. They are named against the seam they sit on so they can be argued with.

CadenceWhen to re-run, and what it costs

The guardrail scan is a MANUAL step, run before each spend, not a hook. It ran before every paid run and passed at 0 each time. The injection probe is a ONE-OFF: it measured 0.0 pct suppression on 67 readings on 2026-08-25 and nothing re-runs it, so that figure ages from the day it was taken.

In one lineNo framework, deliberately

NO FRAMEWORK. A folder of readable Python, no framework dependency, and a prompt anyone can read end to end. A framework abstraction would own the retrieval step -- there is none here -- and the memory layer, which is already the entire surface of src/state.py.

The mappingSeam by seam, who owns what

The seams are the ones this kit already publishes, so the mapping cannot drift from the code.

GraphsWhere LangGraph changes the answer

Linear: src/watch.py -> src/prompt.py -> src/adapters -> evals/scoring.py. No branching, no tool-calling, no agent loop.

The other sideWhat a framework costs you

• Swapping providers means editing the PROVIDERS dict by hand. In exchange the prompt has no hidden templating layer and requirements.txt stays empty.

In one lineWhat one run of this kit actually cost, in time and tokens

Measured on r001-entitlement-sod on the fast tier, with the carried state, 2026-08-25. This kit records telemetry measured per run — 4 of the 6 readings on this axis, 4 of them with a band the guardrail board can judge against.

The readingWhat the last run measured

No movement column. Not one of the 4 earlier runs on record is comparable with this one — a different model tier, or a changed guard — and differencing across that measures two systems rather than one. Said once here rather than as a dash on every row.

Retrieval not shown. No run of this kit has ever recorded a retrieval timing, because the pipeline has no such stage. Those readings are left out rather than dashed: a dash says “this run did not measure it”, which promises a later run could.

DriftMedian call, run over run

Hatched bars are runs that are not comparable with the latest — a different model tier or a changed guard. They are drawn because hiding them would make the history look shorter than it is, and greyed because differencing across them would be measuring two systems.

3 runs not plotted. b000-entitlement-sod-recordtagsonly, b001-entitlement-sod-recordtagsmem, b002-entitlement-sod-structuredlogstrong recorded no call latency at all — a rules-only floor run makes no model call, and the 0 its record stores is an absence, not a zero-millisecond answer. Plotting it would put a bar on this chart claiming the fastest run in the kit’s history.

The other sideWhat this does not tell you

The first of those is the one worth doing something about, and it is a solved problem: instrumenting a pipeline so each step reports its own duration is a couple of lines. Observability / tracing has the code, and what the result looks like once it is running.

In one lineWhat you provision, honestly

Everything measured on this page says where it was measured; everything not measured says that — the same contract as the other thirteen pages. Where a retrieval index would be on other kits, this one has: no index — the population is re-read whole on each scheduled run, and the prior run's carried state is what makes a reading a CHANGE.

The machineWhat this needs

Enumerated from the kit’s own source by build/facts/envscan.py, committed, and drift-checked — never typed into this page.

Your dataWhere everything sits

The wireWhat crosses it, and when

Zero-egress topology: point BASE_URL at a local OpenAI-compatible server and nothing leaves the machine — statically true of the shipped adapter, whose only credential travels in the Authorization header.

The shapeHow it ships, and what you add

The ladderEach decision, and what you provision past its ceiling

Every threshold is a measurement with its provenance beside it, or says is_measured:false. A ceiling with no seam says so — it is the point the kit is outgrown, never a forced join.

When it failsFailure signatures

The knobsThe config surface, enumerated

Environment variables: API_KEY, BASE_URL, EMBED_MODEL, EVAL_WORKERS, MAX_CALLS_PER_DAY, MODEL, PORT, PROVIDER, RECORDED_RUN.

The other sideWhat this page does not claim

The corpus licence, from the Data lens: MIT Your corpus’s licence is yours to verify, and the labels you write are about your documents.

In one lineThe verdict, rule, origin, manager and action call, per reading, exact match against the computed answer key

whether each of the five answered fields equals the answer key, per reading

MethodHow the test was run

python3 -m evals.run --run-id <id> ...; evals/scoring.py compares strings. No model grades anything.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

accuracy = hits / 120 per field. The SoD discriminator and the ACCUMULATED recall are scored separately over their own denominators.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the computed answer key in data/gold.jsonl, produced by src/entitlement.step() from the same planted inputs the corpus builder wrote onto the page.

No true/false rates for this grader. It records 5 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No threshold was swept: exact match has no tunable. Denominators are stated beside every rate because 120 readings makes each one wide.

Cadence: once per run; free

The decisionWhen to reach for it

In one lineA toxic combination held, scored as its own binary over all 120 readings

whether the reading carries a live SoD conflict, apart from the seven-way verdict

MethodHow the test was run

evals/scoring.py, same pass as the exact-match grader.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

caught = readings where (answer implies a conflict) equals (gold implies one), over all 120. The two error directions are reported over their own denominators: 57 readings carrying a live conflict and 63 clean ones.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the computed answer key in data/gold.jsonl, produced by src/entitlement.step() from the same planted inputs the corpus builder wrote onto the page.

No true/false rates for this grader. It records 2 operating rows and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No band finer than one row of the positive class: 57 conflict readings means one miss moves the rate by 1.75 points.

Cadence: once per run; free

The decisionWhen to reach for it

In one lineConflicts whose two halves were granted in DIFFERENT cycles, caught

whether the arm found a conflict that no single cycle's export contains

MethodHow the test was run

evals/scoring.py, same pass.

Every grader on these pages scored the same 120 already-recorded answers. Nothing was re-generated, so this compares rulers and not models.

The inputOne real row, seen by every grader

The formulaWhat it computes

caught / 16 readings whose gold origin is ACCUMULATED.

The analysisWhat it actually did

In operationWhat to monitor

Reference standard: the computed answer key in data/gold.jsonl, produced by src/entitlement.step() from the same planted inputs the corpus builder wrote onto the page.

No true/false rates for this grader. It records 1 operating row and not one of them carries a confusion matrix, so there is no TPR, TNR or precision to state here — this grader does not sort answers into pass and fail, and rates that were never measured are not going to be printed as though they were. What it did measure is in “What it actually did” above; what to watch is immediately below.

How tight can the band be? No sweep: origin is an enum the arm emits, not a thresholded score. 16 cells is a small denominator and the page says so.

Cadence: once per run; free

The decisionWhen to reach for it